Method for sub-pixel value interpolation

ABSTRACT

A method of interpolation in video coding in which an image comprising pixels arranged in rows and columns and represented by values having a specified dynamic range, the pixels in the rows residing at unit horizontal locations and the pixels in the columns residing at unit vertical locations, is interpolated to generate values for sub-pixels at fractional horizontal and vertical locations, the method comprising: a) when values for sub-pixels at half unit horizontal and unit vertical locations, and unit horizontal and half unit vertical locations are required, interpolating such values directly using weighted sums of pixels residing at unit horizontal and unit vertical locations; b) when values for sub-pixels at half unit horizontal and half unit vertical locations are required, interpolating such values directly using a weighted sum of values for sub-pixels residing at half unit horizontal and unit vertical locations calculated according to step (a); and c) when values for sub-pixels at quarter unit horizontal and quarter unit vertical locations are required, interpolating such values by taking the average of at least one pair of a first pair of values of a sub-pixel located at a half unit horizontal and unit vertical location, and a sub-pixel located at a unit horizontal and half unit vertical location and a second pair of values of a pixel located at a unit horizontal and unit vertical location, and a sub-pixel located at a half unit horizontal and half unit vertical location.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to, U.S.application Ser. No. 11/090,717, filed Mar. 25, 2005, now allowed, whichis a continuation of, and claims priority to, U.S. application Ser. No.09/954,608, filed Sep. 17, 2001, now U.S. Pat. No. 6,950,469, all ofwhich are incorporated by reference herein in their entirety.

The present invention relates to a method for sub-pixel valueinterpolation in the encoding and decoding of data. It relatesparticularly, but not exclusively, to encoding and decoding of digitalvideo.

BACKGROUND OF THE INVENTION

Digital video sequences, like ordinary motion pictures recorded on film,comprise a sequence of still images, the illusion of motion beingcreated by displaying the images one after the other at a relativelyfast frame rate, typically 15 to 30 frames per second. Because of therelatively fast frame rate, images in consecutive frames tend to bequite similar and thus contain a considerable amount of redundantinformation. For example, a typical scene may comprise some stationaryelements, such as background scenery, and some moving areas, which maytake many different forms, for example the face of a newsreader, movingtraffic and so on. Alternatively, the camera recording the scene mayitself be moving, in which case all elements of the image have the samekind of motion. In many cases, this means that the overall changebetween one video frame and the next is rather small. Of course, thisdepends on the nature of the movement. For example, the faster themovement, the greater the change from one frame to the next. Similarly,if a scene contains a number of moving elements, the change from oneframe to the next is likely to be greater than in a scene where only oneelement is moving.

It should be appreciated that each frame of a raw, that is uncompressed,digital video sequence comprises a very large amount of imageinformation. Each frame of an uncompressed digital video sequence isformed from an array of image pixels. For example, in a commonly useddigital video format, known as the Quarter Common Interchange Format(QCIF), a frame comprises an array of 176×144 pixels, in which case eachframe has 25,344 pixels. In turn, each pixel is represented by a certainnumber of bits, which carry information about the luminance and/orcolour content of the region of the image corresponding to the pixel.Commonly, a so-called YUV colour model is used to represent theluminance and chrominance content of the image. The luminance, or Y,component represents the intensity (brightness) of the image, while thecolour content of the image is represented by two chrominancecomponents, labelled U and V.

Colour models based on a luminance/chrominance representation of imagecontent provide certain advantages compared with colour models that arebased on a representation involving primary colours (that is Red, Greenand Blue, RGB). The human visual system is more sensitive to intensityvariations than it is to colour variations; YUV colour models exploitthis property by using a lower spatial resolution for the chrominancecomponents (U, V) than for the luminance component (Y). In this way theamount of information needed to code the colour information in an imagecan be reduced with an acceptable reduction in image quality.

The lower spatial resolution of the chrominance components is usuallyattained by sub-sampling. Typically, a block of 16×16 image pixels isrepresented by one block of 16×16 pixels comprising luminanceinformation and the corresponding chrominance components are eachrepresented by one block of 8×8 pixels representing an area of the imageequivalent to that of the 16×16 pixels of the luminance component. Thechrominance components are thus spatially sub-sampled by a factor of 2in the x and y directions. The resulting assembly of one 16×16 pixelluminance block and two 8×8 pixel chrominance blocks is commonlyreferred to as a YUV macroblock, or macroblock, for short.

A QCIF image comprises 11×9 macroblocks. If the luminance blocks andchrominance blocks are represented with 8 bit resolution (that is bynumbers in the range 0 to 255), the total number of bits required permacroblock is (16×16×8)+2×(8×8×8)=3072 bits. The number of bits neededto represent a video frame in QCIF format is thus 99×3072=304,128 bits.This means that the amount of data required to transmit/record/display avideo sequence in QCIF format, represented using a YUV colour model, ata rate of 30 frames per second, is more than 9 Mbps (million bits persecond). This is an extremely high data rate and is impractical for usein video recording, transmission and display applications because of thevery large storage capacity, transmission channel capacity and hardwareperformance required.

If video data is to be transmitted in real-time over a fixed linenetwork such as an ISDN (Integrated Services Digital Network) or aconventional PSTN (Public Service Telephone Network), the available datatransmission bandwidth is typically of the order of 64 kbits/s. Inmobile videotelephony, where transmission takes place at least in partover a radio communications link, the available bandwidth can be as lowas 20 kbits/s. This means that a significant reduction in the amount ofinformation used to represent video data must be achieved in order toenable transmission of digital video sequences over low bandwidthcommunication networks. For this reason video compression techniqueshave been developed which reduce the amount of information transmittedwhile retaining an acceptable image quality.

Video compression methods are based on reducing the redundant andperceptually irrelevant parts of video sequences. The redundancy invideo sequences can be categorised into spatial, temporal and spectralredundancy. ‘Spatial redundancy’ is the term used to describe thecorrelation between neighbouring pixels within a frame. The term‘temporal redundancy’ expresses the fact that the objects appearing inone frame of a sequence are likely to appear in subsequent frames, while‘spectral redundancy’ refers to the correlation between different colourcomponents of the same image.

Sufficiently efficient compression cannot usually be achieved by simplyreducing the various forms of redundancy in a given sequence of images.Thus, most current video encoders also reduce the quality of those partsof the video sequence which are subjectively the least important. Inaddition, the redundancy of the compressed video bit-stream is itselfreduced by means of efficient loss-less encoding. Typically, this isachieved using a technique known as ‘variable length coding’ (VLC).

Modern video compression standards, such as ITU-T recommendations H.261,H.263(+)(++), H.26L and the Motion Picture Experts Group recommendationMPEG-4 make use of ‘motion compensated temporal prediction’. This is aform of temporal redundancy reduction in which the content of some(often many) frames in a video sequence is ‘predicted’ from other framesin the sequence by tracing the motion of objects or regions of an imagebetween frames.

Compressed images which do not make use of temporal redundancy reductionare usually called INTRA-coded or I-frames, whereas temporally predictedimages are called INTER-coded or P-frames. In the case of INTER frames,the predicted (motion-compensated) image is rarely precise enough torepresent the image content with sufficient quality, and therefore aspatially compressed prediction error (PE) frame is also associated witheach INTER frame. Many video compression schemes can also make use ofbi-directionally predicted frames, which are commonly referred to asB-pictures or B-frames. B-pictures are inserted between reference orso-called ‘anchor’ picture pairs (I or P frames) and are predicted fromeither one or both of the anchor pictures. B-pictures are not themselvesused as anchor pictures, that is no other frames are predicted fromthem, and therefore, they can be discarded from the video sequencewithout causing deterioration in the quality of future pictures.

The different types of frame that occur in a typical compressed videosequence are illustrated in FIG. 3 of the accompanying drawings. As canbe seen from the figure, the sequence starts with an INTRA or I frame30. In FIG. 3, arrows 33 denote the ‘forward’ prediction process bywhich P-frames (labelled 34) are formed. The bi-directional predictionprocess by which B-frames (36) are formed is denoted by arrows 31 a and31 b, respectively.

A schematic diagram of an example video coding system using motioncompensated prediction is shown in FIGS. 1 and 2. FIG. 1 illustrates anencoder 10 employing motion compensation and FIG. 2 illustrates acorresponding decoder 20. The encoder 10 shown in FIG. 1 comprises aMotion Field Estimation block 11, a Motion Field Coding block 12, aMotion Compensated Prediction block 13, a Prediction Error Coding block14, a Prediction Error Decoding block 15, a Multiplexing block 16, aFrame Memory 17, and an adder 19. The decoder 20 comprises a MotionCompensated Prediction block 21, a Prediction Error Decoding block 22, aDemultiplexing block 23 and a Frame Memory 24.

The operating principle of video coders using motion compensation is tominimise the amount of information in a prediction error frameE_(n)(x,y), which is the difference between a current frame I_(n)(x,y)being coded and a prediction frame P_(n)(x,y). The prediction errorframe is thus:E _(n)(x,y)=I _(n)(x,y)−P _(n)(x,y).  (1)

The prediction frame P_(n)(x,y) is built using pixel values of areference frame R_(n)(x,y), which is generally one of the previouslycoded and transmitted frames, for example the frame immediatelypreceding the current frame and is available from the Frame Memory 17 ofthe encoder 10. More specifically, the prediction frame P_(n)(x,y) isconstructed by finding so-called ‘prediction pixels’ in the referenceframe R_(n)(x,y) which correspond substantially with pixels in thecurrent frame. Motion information, describing the relationship (e.g.relative location, rotation, scale etc.) between pixels in the currentframe and their corresponding prediction pixels in the reference frameis derived and the prediction frame is constructed by moving theprediction pixels according to the motion information. In this way, theprediction frame is constructed as an approximate representation of thecurrent frame, using pixel values in the reference frame. The predictionerror frame referred to above therefore represents the differencebetween the approximate representation of the current frame provided bythe prediction frame and the current frame itself. The basic advantageprovided by video encoders that use motion compensated prediction arisesfrom the fact that a comparatively compact description of the currentframe can be obtained by representing it in terms of the motioninformation required to form its prediction together with the associatedprediction error information in the prediction error frame.

However, due to the very large number of pixels in a frame, it isgenerally not efficient to transmit separate motion information for eachpixel to the decoder. Instead, in most video coding schemes, the currentframe is divided into larger image segments S_(k) and motion informationrelating to the segments is transmitted to the decoder. For example,motion information is typically provided for each macroblock of a frameand the same motion information is then used for all pixels within themacroblock. In some video coding standards, such as H.26L, a macroblockcan be divided into smaller blocks, each smaller block being providedwith its own motion information.

The motion information usually takes the form of motion vectors[Δx(x,y),Δy(x,y)]. The pair of numbers Δx(x,y) and Δy(x,y) representsthe horizontal and vertical displacements of a pixel at location (x,y)in the current frame I_(n)(x,y) with respect to a pixel in the referenceframe R_(n)(x,y). The motion vectors [Δx(x,y),Δy(x,y)] are calculated inthe Motion Field Estimation block 11 and the set of motion vectors ofthe current frame [Δx(·),Δy(·)] is referred to as the motion vectorfield.

Typically, the location of a macroblock in a current video frame isspecified by the (x,y) co-ordinate of its upper left-hand corner. Thus,in a video coding scheme in which motion information is associated witheach macroblock of a frame, each motion vector describes the horizontaland vertical displacement Δx(x,y) and Δy(x,y) of a pixel representingthe upper left-hand corner of a macroblock in the current frameI_(n)(x,y) with respect to a pixel in the upper left-hand corner of asubstantially corresponding block of prediction pixels in the referenceframe R_(n)(x,y) (as shown in FIG. 4 b).

Motion estimation is a computationally intensive task. Given a referenceframe R_(n)(x,y) and, for example, a square macroblock comprising N×Npixels in a current frame (as shown in FIG. 4 a), the objective ofmotion estimation is to find an N×N pixel block in the reference framethat matches the characteristics of the macroblock in the currentpicture according to some criterion. This criterion can be, for example,a sum of absolute differences (SAD) between the pixels of the macroblockin the current frame and the block of pixels in the reference frame withwhich it is compared. This process is known generally as ‘blockmatching’. It should be noted that, in general, the geometry of theblock to be matched and that in the reference frame do not have to bethe same, as real-world objects can undergo scale changes, as well asrotation and warping. However, in current international video codingstandards, only a translational motion model is used (see below) andthus fixed rectangular geometry is sufficient.

Ideally, in order to achieve the best chance of finding a match, thewhole of the reference frame should be searched. However, this isimpractical as it imposes too high a computational burden on the videoencoder. Instead, the search region is restricted to region [−p,p]around the original location of the macroblock in the current frame, asshown in FIG. 4 c.

In order to reduce the amount of motion information to be transmittedfrom the encoder 10 to the decoder 20, the motion vector field is codedin the Motion Field Coding block 12 of the encoder 10, by representingit with a motion model. In this process, the motion vectors of imagesegments are re-expressed using certain predetermined functions or, inother words, the motion vector field is represented with a model. Almostall currently used motion vector field models are additive motionmodels, complying with the following general formula: $\begin{matrix}{{\Delta\quad{x\left( {x,y} \right)}} = {\sum\limits_{i = 0}^{N - 1}{a_{i}{f_{i}\left( {x,y} \right)}}}} & (2) \\{{\Delta\quad{y\left( {x,y} \right)}} = {\sum\limits_{i = 0}^{M - 1}{b_{i}{g_{i}\left( {x,y} \right)}}}} & (3)\end{matrix}$where coefficients a_(i) and b_(i) are called motion coefficients. Themotion coefficients are transmitted to the decoder 20 (informationstream 2 in FIGS. 1 and 2). Functions f_(i) and g_(i) are called motionfield basis functions, and are known both to the encoder and decoder. Anapproximate motion vector field ({tilde over (Δ)}x(x,y),{tilde over(Δ)}y(x,y)) can be constructed using the coefficients and the basisfunctions. As the basis functions are known to (that is stored in) boththe encoder 10 and the decoder 20, only the motion coefficients need tobe transmitted to the encoder, thus reducing the amount of informationrequired to represent the motion information of the frame.

The simplest motion model is the translational motion model whichrequires only two coefficients to describe the motion vectors of eachsegment. The values of motion vectors are given by:Δx(x,y)=a ₀Δy(x,y)=b ₀  (4)

This model is widely used in various international standards (ISOMPEG-1, MPEG-2, MPEG-4, ITU-T Recommendations H.261 and H.263) todescribe the motion of 16×16 and 8×8 pixel blocks. Systems which use atranslational motion model typically perform motion estimation at fullpixel resolution or some integer fraction of full pixel resolution, forexample at half or one quarter pixel resolution.

The prediction frame P_(n)(x,y) is constructed in the Motion CompensatedPrediction block 13 in the encoder 10, and is given by:P _(n)(x,y)=R _(n) [x+{tilde over (Δ)}x(x,y),y+{tilde over(Δ)}y(x,y)]  (5)

In the Prediction Error Coding block 14, the prediction error frameE_(n)(x,y) is typically compressed by representing it as a finite series(transform) of some 2-dimensional functions. For example, a2-dimensional Discrete Cosine Transform (DCT) can be used. The transformcoefficients are quantised and entropy (for example Huffman) codedbefore they are transmitted to the decoder (information stream 1 inFIGS. 1 and 2). Because of the error introduced by quantisation, thisoperation usually produces some degradation (loss of information) in theprediction error frame E_(n)(x,y). To compensate for this degradation,the encoder 10 also comprises a Prediction Error Decoding block 15,where a decoded prediction error frame {tilde over (E)}_(n)(x,y) isconstructed using the transform coefficients. This locally decodedprediction error frame is added to the prediction frame P_(n)(x,y) inthe adder 19 and the resulting decoded current frame Ĩ_(n)(x,y) isstored in the Frame Memory 17 for further use as the next referenceframe R_(n+1)(x,y).

The information stream 2 carrying information about the motion vectorsis combined with information about the prediction error in multiplexer16 and an information stream 3 containing typically at least those twotypes of information is sent to the decoder 20.

The operation of a corresponding video decoder 20 will now be described.

The Frame Memory 24 of the decoder 20 stores a previously reconstructedreference frame R_(n)(x,y). The prediction frame P_(n)(x,y) isconstructed in the Motion Compensated Prediction block 21 of the decoder20 according to equation 5, using received motion coefficientinformation and pixel values of the previously reconstructed referenceframe R_(n)(x,y). The transmitted transform coefficients of theprediction error frame E_(n)(x,y) are used in the Prediction ErrorDecoding block 22 to construct the decoded prediction error frame {tildeover (E)}_(n)(x,y). The pixels of the decoded current frame Ĩ_(n)(x,y)are then reconstructed by adding the prediction frame P_(n)(x,y) and thedecoded prediction error frame {tilde over (E)}_(n)(x,y):Ĩ _(n)(x,y)=P _(n)(x,y)+{tilde over (E)} _(n)(x,y)=R _(n) [x+{tilde over(Δ)}x(x,y),y+{tilde over (Δ)}y(x,y)]+{tilde over (E)} _(n)(x,y).  (6)

This decoded current frame may be stored in the Frame Memory 24 as thenext reference frame R_(n+1)(x,y).

In the description of motion compensated encoding and decoding ofdigital video presented above, the motion vector [Δx(x,y),Δy(x,y)]describing the motion of a macroblock in the current frame with respectto the reference frame R_(n)(x,y) can point to any of the pixels in thereference frame. This means that motion between frames of a digitalvideo sequence can only be represented at a resolution which isdetermined by the image pixels in the frame (so-called full pixelresolution). Real motion, however, has arbitrary precision, and thus thesystem described above can only provide approximate modelling of themotion between successive frames of a digital video sequence. Typically,modelling of motion between video frames with full pixel resolution isnot sufficiently accurate to allow efficient minimisation of theprediction error (PE) information associated with each macroblock/frame.Therefore, to enable more accurate modelling of real motion and to helpreduce the amount of PE information that must be transmitted fromencoder to decoder, many video coding standards, such as H.263(+)(++)and H.26L, allow motion vectors to point ‘in between’ image pixels. Inother words, the motion vectors can have ‘sub-pixel’ resolution.Allowing motion vectors to have sub-pixel resolution adds to thecomplexity of the encoding and decoding operations that must beperformed, so it is still advantageous to limit the degree of spatialresolution a motion vector may have. Thus, video coding standards, suchas those previously mentioned, typically only allow motion vectors tohave full-, half- or quarter-pixel resolution.

Motion estimation with sub-pixel resolution is usually performed as atwo-stage process, as illustrated in FIG. 5, for a video coding schemewhich allows motion vectors to have full- or half-pixel resolution. Inthe first step, a motion vector having full-pixel resolution isdetermined using any appropriate motion estimation scheme, such as theblock-matching process described in the foregoing. The resulting motionvector, having full-pixel resolution is shown in FIG. 5.

In the second stage, the motion vector determined in the first stage isrefined to obtain the desired half-pixel resolution. In the exampleillustrated in FIG. 5, this is done by forming eight new search blocksof 16×16 pixels, the location of the top-left corner of each block beingmarked with an X in FIG. 5. These locations are denoted as[Δx+m/2,Δy+n/2], where m and n can take the values −1, 0 and +1, butcannot be zero at the same time. As only the pixel values of originalimage pixels are known, the values (for example luminance and/orchrominance values) of the sub-pixels residing at half-pixel locationsmust be estimated for each of the eight new search blocks, using someform of interpolation scheme.

Having interpolated the values of the sub-pixels at half-pixelresolution, each of the eight search blocks is compared with themacroblock whose motion vector is being sought. As in the block matchingprocess performed in order to determine the motion vector with fullpixel resolution, the macroblock is compared with each of the eightsearch blocks according to some criterion, for example a SAD. As aresult of the comparisons, a minimum SAD value will generally beobtained. Depending on the nature of the motion in the video sequence,this minimum value may correspond to the location specified by theoriginal motion vector (having full-pixel resolution), or it maycorrespond to a location having a half-pixel resolution. Thus, it ispossible to determine whether a motion vector should point to afull-pixel or sub-pixel location and if sub-pixel resolution isappropriate, to determine the correct sub-pixel resolution motionvector. It should also be appreciated that the scheme just described canbe extended to other sub-pixel resolutions (for example,one-quarter-pixel resolution) in an entirely analogous fashion.

In practice, the estimation of a sub-pixel value in the reference frameis performed by interpolating the value of the sub-pixel fromsurrounding pixel values. In general, interpolation of a sub-pixel valueF(x,y) situated at a non-integer location (x,y)=(n+Δx, m+Δy), can beformulated as a two-dimensional operation, represented mathematicallyas: $\begin{matrix}{{F\left( {x,y} \right)} = {\sum\limits_{k = {- K}}^{K = 1}{\sum\limits_{l = {- L}}^{L = 1}{{f\left( {{k + K},{l + L}} \right)}{F\left( {{n + k},{m + l}} \right)}}}}} & (7)\end{matrix}$where f(k,l) are filter coefficients and n and m are obtained bytruncating x and y, respectively, to integer values. Typically, thefilter coefficients are dependent on the x and y values and theinterpolation filters are usually so-called ‘separable filters’, inwhich case sub-pixel value F(x,y) can be calculated as follows:$\begin{matrix}{{F\left( {x,y} \right)} = {\sum\limits_{k = {- K}}^{K = 1}{{f\left( {k + K} \right)}{\sum\limits_{l = {- K}}^{K = 1}{{f\left( {l + K} \right)}{F\left( {{n + k},{m + l}} \right)}}}}}} & (8)\end{matrix}$

The motion vectors are calculated in the encoder. Once the correspondingmotion coefficients are transmitted to the decoder, it is astraightforward matter to interpolate the required sub-pixels using aninterpolation method identical to that used in the encoder. In this way,a frame following a reference frame in the Frame Memory 24, can bereconstructed from the reference frame and the motion vectors.

The simplest way of applying sub-pixel value interpolation in a videocoder is to interpolate each sub-pixel value every time it is needed.However, this is not an efficient solution in a video encoder, becauseit is likely that the same sub-pixel value will be required severaltimes and thus calculations to interpolate the same sub-pixel value willbe performed multiple times. This results in an unnecessary increase ofcomputational complexity/burden in the encoder.

An alternative approach, which limits the complexity of the encoder, isto pre-calculate and store all sub-pixel values in a memory associatedwith the encoder. This solution is called interpolation ‘before-hand’interpolation hereafter in this document. While limiting complexity,before-hand interpolation has the disadvantage of increasing memoryusage by a large margin. For example, if the motion vector accuracy isone quarter pixel in both horizontal and vertical dimensions, storingpre-calculated sub-pixel values for a complete image results in a memoryusage that is 16 times that required to store the original,non-interpolated image. In addition, it involves the calculation of somesub-pixels which might not actually be required in calculating motionvectors in the encoder. Before-hand interpolation is also particularlyinefficient in a video decoder, as the majority of pre-calculatedsub-pixel values will never be required by the decoder. Thus, it isadvantageous not to use pre-calculation in the decoder.

So-called ‘on-demand’ interpolation can be used to reduce memoryrequirements in the encoder. For example, if the desired pixel precisionis quarter pixel resolution, only sub-pixels at one half unit resolutionare interpolated before-hand for the whole frame and stored in thememory. Values of one-quarter pixel resolution sub-pixels are onlycalculated during the motion estimation/compensation process as and whenit is required. In this case memory usage is only 4 times that requiredto store the original, non-interpolated image.

It should be noted that when before-hand interpolation is used, theinterpolation process constitutes only a small fraction of the totalencoder computational complexity/burden, since every pixel isinterpolated just once. Therefore, in the encoder, the complexity of theinterpolation process itself is not very critical when before-handsub-pixel value interpolation is used. On the other hand, on-demandinterpolation poses a much higher computational burden on the encoder,since sub-pixels may be interpolated many times. Hence the complexity ofinterpolation process, which may be considered in terms of the number ofcomputational operations or operational cycles that must be performed inorder to interpolate the sub-pixel values, becomes an importantconsideration.

In the decoder, the same sub-pixel values are used a few times at mostand some are not needed at all. Therefore, in the decoder it isadvantageous not to use before-hand interpolation at all, that is, it isadvantageous not to pre-calculate any sub-pixel values.

Two interpolation schemes have been developed as part of the workongoing in the ITU-Telecommunications Standardization Sector, StudyGroup 16, Video Coding Experts Group (VCEG), Questions 6 and 15. Theseapproaches were proposed for incorporation into ITU-T recommendationH.26L and have been implemented in test models (TML) for the purposes ofevaluation and further development. The test model corresponding toQuestion 15 is referred to as Test Model 5 (TML5), while that resultingfrom Question 6 is known as Test Model 6 (TML6). The interpolationschemes proposed in both TML5 and TML6 will now be described.

Throughout the description of the sub-pixel value interpolation schemeused in test model TML5, reference will be made to FIG. 12 a, whichdefines a notation for describing pixel and sub-pixel locations specificto TML5. A separate notation, defined in FIG. 13 a, will be used in thediscussion of the sub-pixel value interpolation scheme used in TML6. Astill further notation, illustrated in FIG. 14 a, will be used later inthe text in connection with the sub-pixel value interpolation methodaccording to the invention. It should be appreciated that the threedifferent notations used in the text are intended to assist in theunderstanding of each interpolation method and to help distinguishdifferences between them. However, in all three figures, the letter A isused to denote original image pixels (full pixel resolution). Morespecifically, the letter A represents the location of pixels in theimage data representing a frame of a video sequence, the pixel values ofpixels A being either received as current frame I_(n)(x,y) from a videosource, or reconstructed and stored as a reference frame R_(n)(x,y) inthe Frame Memory 17, 24 of the encoder 10 or the decoder 20. All otherletters represent sub-pixel locations, the values of the sub-pixelssituated at the sub-pixel locations being obtained by interpolation.

Certain other terms will also be used in a consistent manner throughoutthe text to identify particular pixel and sub-pixel locations. These areas follows:

The term ‘unit horizontal location’ is used to describe the location ofany sub-pixel that is constructed in a column of the original imagedata. Sub-pixels c and e in FIGS. 12 a and 13 a, as well as sub-pixels band e in FIG. 14 a have unit horizontal locations.

The term ‘unit vertical location’ is used to describe any sub-pixel thatis constructed in a row of the original image data. Sub-pixels b and din FIGS. 12 a and 13 a as well as sub-pixels b and d in FIG. 14 a haveunit vertical locations.

By definition, pixels A have unit horizontal and unit verticallocations.

The term ‘half horizontal location’ is used to describe the location ofany sub-pixel that is constructed in a column that lies at half pixelresolution. Sub-pixels b, c, and e shown in FIGS. 12 a and 13 a fallinto this category, as do sub-pixels b, c and f in FIG. 14 a. In asimilar manner, the term ‘half vertical location’ is used to describethe location of any sub-pixel that is constructed in a row that lies athalf-pixel resolution, such as sub-pixels c and d in FIGS. 12 a and 13a, as well as sub-pixels b, c and g in FIG. 14 a.

Furthermore, the term ‘quarter horizontal location’ refers to anysub-pixel that is constructed in a column which lies at quarter-pixelresolution, such as sub-pixels d and e in FIG. 12 a, sub-pixels d and gin FIG. 13 a and sub-pixels d, g and h in FIG. 14 a. Analogously, theterm ‘quarter vertical location’ refers to sub-pixels that areconstructed in a row which lies at quarter-pixel resolution. In FIG. 12a, sub-pixels e and f fall into this category, as do sub-pixels e, f andg in FIG. 13 a and sub-pixels e, f and h in FIG. 14 a.

The definition of each of the terms described above is shown by‘envelopes’ drawn on the corresponding figures.

It should further be noted that it is often convenient to denote aparticular pixel with a two-dimensional reference. In this case, theappropriate two-dimensional reference can be obtained by examining theintersection of the envelopes in FIGS. 12 a, 13 a and 14 a. Applyingthis principle, pixel d in FIG. 12 a, for example, has a half horizontaland half vertical location and sub-pixel e has a unit horizontal andquarter vertical location. In addition, and for ease of reference,sub-pixels that reside at half unit horizontal and unit verticallocations, unit horizontal and half unit vertical locations as well ashalf unit horizontal and half unit vertical locations, will be referredto as ½ resolution sub-pixels. Sub-pixels which reside at any quarterunit horizontal and/or quarter unit vertical location will be referredto as ¼ resolution sub-pixels.

It should also be noted that in the descriptions of the two test modelsand in the detailed description of the invention itself, it will beassumed that pixels have a minimum value of 0 and a maximum value of2^(n)−1 where n is the number of bits reserved for a pixel value. Thenumber of bits is typically 8. After a sub-pixel has been interpolated,if the value of that interpolated sub-pixel exceeds the value of2^(n)−1, it is restricted to the range of [0, 2^(n)−1], i.e. valueslower than the minimum allowed value will become the minimum value (0)and values larger than the maximum will the become maximum value(2^(n)−1). This operation is called clipping.

The sub-pixel value interpolation scheme according to TML5 will now bedescribed in detail with reference to FIGS. 12 a, 12 b and 12 c.

-   1. The value for the sub-pixel at half unit horizontal and unit    vertical location, that is ½ resolution sub-pixel b in FIG. 12 a, is    calculated using a 6-tap filter. The filter interpolates a value for    ½ resolution sub-pixel b based upon the values of the 6 pixels (A₁    to A₆) situated in a row at unit horizontal locations and unit    vertical locations symmetrically about b, as shown in FIG. 12 b,    according to the formula b=(A₁−5A₂+20A₃+20A₄−5A₅+A₆+16)/32. The    operator / denotes division with truncation. The result is clipped    to lie in the range [0, 2^(n)−1].-   2. Values for the ½ resolution sub-pixels labelled c are calculated    using the same six tap filter as used in step 1 and the six nearest    pixels or sub-pixels (A or b) in the vertical direction. Referring    now to FIG. 12 c, the filter interpolates a value for the ½    resolution sub-pixel c located at unit horizontal and half vertical    location based upon the values of the 6 pixels (A₁ to A₆) situated    in a column at unit horizontal locations and unit vertical locations    symmetrically about c, according to the formula    c=(A₁−5A₂+20A₃+20A₄−5A₅+A₆+16)/32. Similarly, a value for the ½    resolution sub-pixel c at half horizontal and half vertical location    is calculated according to c=(b₁−5b₂+20b₃+20b₄−5b₅+b₆+16)/32. Again,    the operator / denotes division with truncation. The values    calculated for the c sub-pixels are further clipped to lie in the    range [0, 2^(n)−1].

At this point in the interpolation process the values of all ½resolution sub-pixels have been calculated and the process proceeds tothe calculation of ¼ resolution sub-pixel values.

-   3. Values for the ¼ resolution sub-pixels labelled d are calculated    using linear interpolation and the values of the nearest pixels    and/or ½ resolution sub-pixels in the horizontal direction. More    specifically, values for ¼ resolution sub-pixels d located at    quarter horizontal and unit vertical locations, are calculated by    taking the average of the immediately neighbouring pixel at unit    horizontal and unit vertical location (pixel A) and the immediately    neighbouring ½ resolution sub-pixel at half horizontal and unit    vertical location (sub-pixel b), i.e. according to d=(A+b)/2. Values    for ¼ resolution sub-pixels d located at quarter horizontal and half    vertical locations, are calculated by taking the average of the    immediately neighbouring ½ resolution sub-pixels c which lie at unit    horizontal and half vertical location and half horizontal and half    vertical locations respectively, i.e. according to d=(c₁+c₂)/2.    Again operator / indicates division with truncation.-   4. Values for the ¼ resolution sub-pixels labelled e are calculated    using linear interpolation and the values of the nearest pixels    and/or ½ resolution sub-pixels in the vertical direction. In    particular, ¼ resolution sub-pixels e at unit horizontal and quarter    vertical locations are calculated by taking the average of the    immediately neighbouring pixel at unit horizontal and unit vertical    location (pixel A) and the immediately neighbouring sub-pixel at    unit horizontal and half vertical location (sub-pixel c) according    to e=(A+c)/2. ¼ resolution sub-pixels e₃ at half horizontal and    quarter vertical locations are calculated by taking the average of    the immediately neighbouring sub-pixel at half horizontal and unit    vertical location (sub-pixel b) and the immediately neighbouring    sub-pixel at half horizontal and half vertical location (sub-pixel    c), according to e=(b+c)/2. Furthermore, ¼ resolution sub-pixels e    at quarter horizontal and quarter vertical locations are calculated    by taking the average of the immediately neighbouring sub-pixels at    quarter horizontal and unit vertical location and the corresponding    sub-pixel at quarter horizontal and half vertical location    (sub-pixels d), according to e=(d₁+d₂)/2. Once more, operator /    indicates division with truncation.-   5. The value for ¼ resolution sub-pixel f is interpolated by    averaging the values of the 4 closest pixels values at unit    horizontal and vertical locations, according to f=(A₁+A₂+A₃+A₄+2)/4,    where pixels A₁, A₂, A₃ and A₄ are the four nearest original pixels.

A disadvantage of TML5 is that the decoder is computationally complex.This results from the fact that TML5 uses an approach in whichinterpolation of ¼ resolution sub-pixel values depends upon theinterpolation of ½ resolution sub-pixel values. This means that in orderto interpolate the values of the ¼ resolution sub-pixels, the values ofthe ½ resolution sub-pixels from which they are determined must becalculated first. Furthermore, since the values of some of the ¼resolution sub-pixels depend upon the interpolated values obtained forother ¼ resolution sub-pixels, truncation of the ¼ resolution sub-pixelvalues has a deleterious effect on the precision of some of the ¼resolution sub-pixel values. Specifically, the ¼ resolution sub-pixelvalues are less precise than they would be if calculated from valuesthat had not been truncated and clipped. Another disadvantage of TML5 isthat it is necessary to store the values of the ½ resolution sub-pixelsin order to interpolate the ¼ resolution sub-pixel values. Therefore,excess memory is required to store a result which is not ultimatelyrequired.

The sub-pixel value interpolation scheme according to TML6, referred toherein as direct interpolation, will now be described. In the encoderthe interpolation method according to TML6 works like the previouslydescribed TML5 interpolation method, except that maximum precision isretained throughout. This is achieved by using intermediate values whichare neither rounded nor clipped. A step-by-step description ofinterpolation method according to TML6 as applied in the encoder isgiven below with reference to FIGS. 13 a, 13 b and 13 c.

-   1. The value for the sub-pixel at half unit horizontal and unit    vertical location, that is ½ resolution sub-pixel b in FIG. 13 a, is    obtained by first calculating an intermediate value b using a six    tap filter. The filter calculates b based upon the values of the 6    pixels (A₁ to A₆) situated in a row at unit horizontal locations and    unit vertical locations symmetrically about b, as shown in FIG. 13    b, according to the formula b=(A₁−5A₂+20A₃+20A₄−5A₅+A₆). The final    value of b is then calculated as b=(b+16)/32 and is clipped to lie    in the range [0, 2^(n)−1]. As before, the operator / denotes    division with truncation.-   2. Values for the ½ resolution sub-pixels labelled c are obtained by    first calculating intermediate values c. Referring to FIG. 13 c, an    intermediate value c for the ½ resolution sub-pixel c located at    unit horizontal and half vertical location is calculated based upon    the values of the 6 pixels (A₁ to A₆) situated in a column at unit    horizontal locations and unit vertical locations symmetrically about    c, according to the formula c=(A₁−5A₂+20A₃+20A₄−5A₅+A₆). The final    value for the ½ resolution sub-pixel c located at unit horizontal    and half vertical location is then calculated according to    c=(c+16)/32. Similarly, an intermediate value c for the ½ resolution    sub-pixel c at half horizontal and half vertical location is    calculated according to c=(b₁−5b₂+20b₃+20b₄−5b₅+b₆). A final value    for this ½ resolution sub-pixel is then calculated according to    (c+512)/1024. Again, the operator / denotes division with truncation    and the values calculated for ½ resolution sub-pixels c are further    clipped to lie in the range [0, 2^(n)−1].-   3. Values for the ¼ resolution sub-pixels labelled d are calculated    as follows. Values for ¼ resolution sub-pixels d located at quarter    horizontal and unit vertical locations, are calculated from the    value of the immediately neighbouring pixel at unit horizontal and    unit vertical location (pixel A) and the intermediate value b    calculated in step (1) for the immediately neighbouring ½ resolution    sub-pixel at half horizontal and unit vertical location (½    resolution sub-pixel b), according to d=(32A+b+32)/64. Values for ¼    resolution sub-pixels d located at quarter horizontal and half    vertical locations, are interpolated using the intermediate values c    calculated for the immediately neighbouring ½ resolution sub-pixels    c which lie at unit horizontal and half vertical location and half    horizontal and half vertical locations respectively, according to    d=(32c₁, +c₂+1024)/2048. Again operator / indicates division with    truncation and the finally obtained ¼ resolution sub-pixel values d    are clipped to lie in the range [0, 2^(n)−1].-   4. Values for the ¼ resolution sub-pixels labelled e are calculated    as follows. Values for ¼ resolution sub-pixels e located at unit    horizontal and quarter vertical locations are calculated from the    value of the immediately neighbouring pixel at unit horizontal and    unit vertical location (pixel A) and the intermediate value c    calculated in step (2) for the immediately neighbouring ½ resolution    sub-pixel at unit horizontal and half vertical location, according    to e=(32A+c+32)/64. Values for ¼ resolution sub-pixels e located at    half horizontal and quarter vertical locations are calculated from    the intermediate value b calculated in step (1) for the immediately    neighbouring ½ resolution sub-pixel at half horizontal and unit    vertical location and the intermediate value c calculated in    step (2) for the immediately neighbouring ½ resolution sub-pixel at    half horizontal and half vertical location, according to    e=(32b+c+1024)/2048. Once more, operator / indicates division with    truncation and the finally obtained ¼ resolution sub-pixel values e    are clipped to lie in the range [0, 2^(n)−1].-   5. Values for ¼ resolution sub-pixels labelled g are computed using    the value of the nearest original pixel A and the intermediate    values of the three nearest neighbouring ½ resolution sub-pixels,    according to g=(1024A+32b+32c₁+c₂+2048)/4096. As before, operator /    indicates division with truncation and the finally obtained for ¼    resolution sub-pixel values g are clipped to lie in the range [0,    2^(n)−1].-   6. The value for ¼ resolution sub-pixel f is interpolated by    averaging the values of the 4 closest pixels at unit horizontal and    vertical locations, according to f=(A₁+A₂+A₃+A₄₊₂)/4, where the    locations of pixels A₁, A₂, A₃ and A₄ are the four nearest original    pixels.

In the decoder, sub-pixel values can be obtained directly by applying6-tap filters in horizontal and vertical directions. In the case of ¼sub-pixel resolution, referring to FIG. 13 a, the filter coefficientsapplied to pixels and sub-pixels at unit vertical location are [0, 0,64, 0, 0, 0] for a set of six pixels A, [1, −5, 52, 20, −5, 1] for a setof six sub-pixels d, [2, −10, 40, 40, −10, 2] for a set of sixsub-pixels b, and [1, −5, 20, 52, −5, 1] for a set of six sub-pixels d.These filter coefficients are applied to respective sets of pixels orsub-pixels in the same row as the sub-pixel values being interpolated.

After applying the filters in the horizontal and vertical directions,interpolated value c is normalized according to c=(c+2048)/4096 andclipped to lie in the range [0, 2^(n)−1]. When a motion vector points toan integer pixel position in either the horizontal or verticaldirection, many zero coefficients are used. In a practicalimplementation of TML6, different branches are used in the softwarewhich are optimised for the different sub-pixel cases so that there areno multiplications by zero coefficients.

It should be noted that in TML6, ¼ resolution sub-pixel values areobtained directly using the intermediate values referred to above andare not derived from rounded and clipped values for ½ resolutionsub-pixels. Therefore, in obtaining the ¼ resolution sub-pixel values,it is not necessary to calculate final values for any of the ½resolution sub-pixels. Specifically, it is not necessary to carry outthe truncation and clipping operations associated with the calculationof final values for the ½ resolution sub-pixels. Neither is it necessaryto have stored final values for ½ resolution sub-pixels for use in thecalculation of the ¼ resolution sub-pixel values. Therefore TML6 iscomputationally less complex than TML5, as fewer truncating and clippingoperations are required. However, a disadvantage of TML6 is that highprecision arithmetic is required both in the encoder and in the decoder.High precision interpolation requires more silicon area in ASICs andrequires more computations in some CPUs. Furthermore, implementation ofdirect interpolation as specified in TML6 in an on-demand fashion has ahigh memory requirement. This is an important factor, particularly inembedded devices.

In view of the previously presented discussion, it should be appreciatedthat due to the different requirements of the video encoder and decoderwith regard to sub-pixel interpolation, there exists a significantproblem in developing a method of sub-pixel value interpolation capableof providing satisfactory performance in both the encoder and decoder.Furthermore, neither of the current test models (TML5, TML6) describedin the foregoing can provide a solution that is optimum for applicationin both encoder and decoder.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is provided method ofinterpolation in video coding in which an image comprising pixelsarranged in rows and columns and represented by values having aspecified dynamic range, the pixels in the rows residing at unithorizontal locations and the pixels in the columns residing at unitvertical locations, is interpolated to generate values for sub-pixels atfractional horizontal and vertical locations, the fractional horizontaland vertical locations being defined according to ½^(x), where x is apositive integer having a maximum value N, the method comprising:

a) when values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations are required, interpolating such values directly usingweighted sums of pixels residing at unit horizontal and unit verticallocations;

b) when values for sub-pixels at ½^(N−1) unit horizontal and ½^(N−1)unit vertical locations are required, interpolating such values directlyusing a choice of a first weighted sum of values for sub-pixels residingat ½^(N−1) unit horizontal and unit vertical locations and a secondweighted sum of values for sub-pixels residing at unit horizontal and½^(N−1) unit vertical locations, the first and second weighted sums ofvalues being calculated according to step (a); and

c) when a value for a sub-pixel situated at a ½^(N) unit horizontal and½^(N) unit vertical location is required, interpolating such a value bytaking a weighted average of the value of a first sub-pixel or pixelsituated at a ½^(N−m) unit horizontal and ½^(N−n) unit vertical locationand the value of a second sub-pixel or pixel located at a ½^(N−p) unithorizontal and ½^(N−q) unit vertical location, variables m, n, p and qtaking integer values in the range 1 to N such that the first and secondsub-pixels or pixels are located diagonally with respect to thesub-pixel at ½^(N) unit horizontal and ½^(N) vertical location.

Preferably a first and a second weight are used in the weighted averagereferred to in (c), the relative magnitudes of the weights beinginversely proportional to the (straight-line diagonal) proximity of thefirst and the second sub-pixel or pixel to the sub-pixel at ½^(N) unithorizontal and ½^(N) vertical location.

In a situation where the first and the second sub-pixel or pixel aresymmetrically located with respect to (equidistant from) the sub-pixelat ½^(N) unit horizontal and ½^(N) vertical location, the first andsecond weights may have equal values.

The first weighted sum of values for sub-pixels residing at ½^(N−1) unithorizontal and unit vertical locations in step b) may be used when asub-pixel at ½^(N−1) unit horizontal and ½^(N) unit vertical location isrequired.

The second weighted sum of values for sub-pixels residing at unithorizontal and ½^(N−1) unit vertical locations in step b) may be usedwhen a sub-pixel at ½^(N) unit horizontal and ½^(N−1) unit verticallocation is required.

In one embodiment, when values for sub-pixels at ½^(N) unit horizontaland unit vertical locations, and ½^(N) horizontal and ½^(N−1) verticallocations are required, such values are interpolated by taking theaverage of the values of a first pixel or sub-pixel located at avertical location corresponding to that of the sub-pixel beingcalculated and unit horizontal location and a second pixel or sub-pixellocated at a vertical location corresponding to that of the sub-pixelbeing calculated and ½^(N−1) unit horizontal location.

When values for sub-pixels at unit horizontal and ½^(N) unit verticallocations, and ½^(N−1) unit horizontal and ½^(N) unit vertical locationsare required, they may be interpolated by taking the average of thevalues of a first pixel or sub-pixel located at a horizontal locationcorresponding to that of the sub-pixel being calculated and unitvertical location and a second pixel or sub-pixel located at ahorizontal location corresponding to that of the sub-pixel beingcalculated and ½^(N−1) unit vertical location.

Values for sub-pixels at ½^(N) unit horizontal and ½^(N) unit verticallocations may be interpolated by taking the average of values of a pixellocated at a unit horizontal and unit vertical location, and a sub-pixellocated at a ½^(N−1) unit horizontal and ½^(N−1) unit vertical location.

Values for sub-pixels at ½^(N) unit horizontal and ½^(N) unit verticallocations may be interpolated by taking the average of values of asub-pixel located at a ½^(N−1) unit horizontal and unit verticallocation, and a sub-pixel located at a unit horizontal and ½^(N−1) unitvertical location.

Values for half of the sub-pixels at ½^(N) unit horizontal and ½^(N)unit vertical locations may be interpolated by taking the average of afirst pair of values of a sub-pixel located at a ½^(N−1) unit horizontaland unit vertical location, and a sub-pixel located at a unit horizontaland ½^(N−1) unit vertical location and values for the other half of thesub-pixels at ½^(N) unit horizontal and ½^(N) unit vertical locationsare interpolated by taking the average of a second pair of values of apixel located at a unit horizontal and unit vertical location, and asub-pixel located at a ½^(N−1) unit horizontal and ½^(N−1) unit verticallocation.

Values for sub-pixels at ½^(N) unit horizontal and ½^(N) unit verticallocations are alternately interpolated for one such sub-pixel by takingthe average of a first pair of values of a sub-pixel located at a½^(N−1) unit horizontal and unit vertical location, and a sub-pixellocated at a unit horizontal and ½^(N−1) unit vertical location andvalues and for a neighbouring such sub-pixel by taking the average of asecond pair of values of a pixel located at a unit horizontal and unitvertical location, and a sub-pixel located at a ½^(N−1) unit horizontaland ½^(N−1) unit vertical location.

The sub-pixels at ½^(N) unit horizontal and ½^(N) unit verticallocations may be alternately interpolated in a horizontal direction.

The sub-pixels at ½^(N) unit horizontal and ½^(N) unit verticallocations may be alternately interpolated in a horizontal direction.

When values for some sub-pixels at ½^(N) unit horizontal and ½^(N) unitvertical locations are required, such values may be interpolated bytaking the average of a plurality of nearest neighbouring pixels.

At least one of step a) and step b) interpolating sub-pixel valuesdirectly using weighted sums may involve the calculation of anintermediate value for the sub-pixel values having a dynamic rangegreater than the specified dynamic range.

The intermediate value for a sub-pixel having ½^(N−1) sub-pixelresolution may be used the calculation of a sub-pixel value having ½^(N)sub-pixel resolution.

According to a second aspect of the invention, there is provided amethod of interpolation in video coding in which an image comprisingpixels arranged in rows and columns and represented by values having aspecified dynamic range, the pixels in the rows residing at unithorizontal locations and the pixels in the columns residing at unitvertical locations, is interpolated to generate values for sub-pixels atfractional horizontal and vertical locations, the method comprising:

a) when values for sub-pixels at half unit horizontal and unit verticallocations, and unit horizontal and half unit vertical locations arerequired, interpolating such values directly using weighted sums ofpixels residing at unit horizontal and unit vertical locations;

b) when values for sub-pixels at half unit horizontal and half unitvertical locations are required, interpolating such values directlyusing a weighted sum of values for sub-pixels residing at half unithorizontal and unit vertical locations calculated according to step (a);and

c) when values for sub-pixels at quarter unit horizontal and quarterunit vertical locations are required, interpolating such values bytaking the average of at least one pair of a first pair of values of asub-pixel located at a half unit horizontal and unit vertical location,and a sub-pixel located at a unit horizontal and half unit verticallocation and a second pair of values of a pixel located at a unithorizontal and unit vertical location, and a sub-pixel located at a halfunit horizontal and half unit vertical location.

According to a third aspect of the invention, there is provided a methodof interpolation in video coding in which an image comprising pixelsarranged in rows and columns and represented by values having aspecified dynamic range, the pixels in the rows residing at unithorizontal locations and the pixels in the columns residing at unitvertical locations, is interpolated to generate values for sub-pixels atfractional horizontal and vertical locations, the fractional horizontaland vertical locations being defined according to ½^(x) where x is apositive integer having a maximum value N, the method comprising:

a) when values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations are required, interpolating such values directly usingweighted sums of pixels residing at unit horizontal and unit verticallocations;

b) when a value for a sub-pixel at a sub-pixel horizontal and sub-pixelvertical location is required, interpolating such a value directly usinga choice of a first weighted sum of values for sub-pixels located at avertical location corresponding to that of the sub-pixel beingcalculated and a second weighted sum of values for sub-pixels located ata horizontal location corresponding to that of the sub-pixel beingcalculated.

The sub-pixels used in the first weighted sum may be sub-pixels residingat ½^(N−1) unit horizontal and unit vertical locations and the firstweighted sum may be used to interpolate a value for a sub-pixel at½^(N−1) unit horizontal and ½^(N) unit vertical location.

The sub-pixels used in the second weighted sum may be sub-pixelsresiding at unit horizontal and ½^(N−1) unit vertical locations and thesecond weighted sum may be used to interpolate a value for a sub-pixelat ½^(N) unit horizontal and ½^(N−1) unit vertical location.

When values for sub-pixels at ½^(N) unit horizontal and ½^(N) unitvertical locations are required, they may be interpolated by taking theaverage of at least one pair of a first pair of values of a sub-pixellocated at a ½^(N−1) unit horizontal and unit vertical location, and asub-pixel located at a unit horizontal and ½^(N−1) unit verticallocation and a second pair of values of a pixel located at a unithorizontal and unit vertical location, and a sub-pixel located at a½^(N−1) unit horizontal and ½^(N−1) unit vertical location.

In the foregoing, N may be equal an integer selected from a listconsisting of the values 2, 3, and 4.

Sub-pixels at quarter unit horizontal location are to be interpreted asbeing sub-pixels having as their left-hand nearest neighbour a pixel atunit horizontal location and as their right-hand nearest neighbour asub-pixel at half unit horizontal location as well as sub-pixels havingas their left-hand nearest neighbour a sub-pixel at half unit horizontallocation and as their right-hand nearest neighbour a pixel at unithorizontal location. Correspondingly, sub-pixels at quarter unitvertical location are to be interpreted as being sub-pixels having astheir upper nearest neighbour a pixel at unit vertical location and astheir lower nearest neighbour a sub-pixel at half unit vertical locationas well as sub-pixels having as their upper nearest neighbour asub-pixel at half unit vertical location and as their lower nearestneighbour a pixel at unit vertical location.

The term dynamic range, refers to the range of values which thesub-pixel values and the weighted sums can take.

Preferably changing the dynamic range, whether by extending it orreducing it, means changing the number of bits which are used torepresent the dynamic range.

In an embodiment of the invention, the method is applied to an imagethat is sub-divided into a number of image blocks. Preferably each imageblock comprises four corners, each corner being defined by a pixellocated at a unit horizontal and unit vertical location. Preferably themethod is applied to each image block as the block becomes available forsub-pixel value interpolation. Alternatively, sub-pixel valueinterpolation according to the method of the invention is performed onceall image blocks of an image have become available for sub-pixel valueinterpolation.

Preferably the method is used in video encoding. Preferably the methodis used in video decoding.

In one embodiment of the invention, when used in encoding, the method iscarried out as before-hand interpolation, in which values for allsub-pixels at half unit locations and values for all sub-pixels atquarter unit locations are calculated and stored before beingsubsequently used in the determination of a prediction frame duringmotion predictive coding. In alternative embodiments, the method iscarried out as a combination of before-hand and on-demand interpolation.In this case, a certain proportion or category of sub-pixel values iscalculated and stored before being used in the determination of aprediction frame and certain other sub-pixel values are calculated onlywhen required during motion predictive coding.

Preferably, when the method is used in decoding, sub-pixels are onlyinterpolated when their need is indicated by a motion vector.

According to a fourth aspect of the invention, there is provided a videocoder for coding an image comprising pixels arranged in rows and columnsand represented by values having a specified dynamic range, the pixelsin the rows residing at unit horizontal locations and the pixels in thecolumns residing at unit vertical locations, the video coder comprisingan interpolator adapted to generate values for sub-pixels at fractionalhorizontal and vertical locations, the fractional horizontal andvertical locations being defined according to ½^(x), where x is apositive integer having a maximum value N, the interpolator beingadapted to:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate values for sub-pixels at ½^(N−1) unit horizontal and½^(N−1) unit vertical locations directly using a choice of a firstweighted sum of values for sub-pixels residing at ½^(N−1) unithorizontal and unit vertical locations and a second weighted sum ofvalues for sub-pixels residing at unit horizontal and ½^(N−1) unitvertical locations, the first and second weighted sums of values beingcalculated according to step (a); and

c) interpolate a value for a sub-pixel situated at a ½^(N) unithorizontal and ½^(N) unit vertical location by taking a weighted averageof the value of a first sub-pixel or pixel situated at a ½^(N−m) unithorizontal and ½^(N−n) unit vertical location and the value of a secondsub-pixel or pixel located at a ½^(N−p) unit horizontal and ½^(N−q) unitvertical location, variables m, n, p and q taking integer values in therange 1 to N such that the first and second sub-pixels or pixels arelocated diagonally with respect to the sub-pixel at ½^(N) unithorizontal and ½^(N) vertical location.

The video coder may comprise a video encoder. It may comprise a videodecoder. There may be a codec comprising both the video encoder and thevideo decoder.

According to a fifth aspect of the invention, there is provided acommunications terminal comprising a video coder for coding an imagecomprising pixels arranged in rows and columns and represented by valueshaving a specified dynamic range, the pixels in the rows residing atunit horizontal locations and the pixels in the columns residing at unitvertical locations, the video coder comprising an interpolator adaptedto generate values for sub-pixels at fractional horizontal and verticallocations, the fractional horizontal and vertical locations beingdefined according to ½^(x), where x is a positive integer having amaximum value N, the interpolator being adapted to:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate values for sub-pixels at ½^(N−1) unit horizontal and½^(N−1) unit vertical locations directly using a choice of a firstweighted sum of values for sub-pixels residing at ½^(N−1) unithorizontal and unit vertical locations and a second weighted sum ofvalues for sub-pixels residing at unit horizontal and ½^(N−1) unitvertical locations, the first and second weighted sums of values beingcalculated according to step (a); and

c) interpolate a value for a sub-pixel situated at a ½^(N) unithorizontal and ½^(N) unit vertical location by taking a weighted averageof the value of a first sub-pixel or pixel situated at a ½^(N−m) unithorizontal and ½^(N−n) unit vertical location and the value of a secondsub-pixel or pixel located at a ½^(N−p) unit horizontal and ½^(N−q) unitvertical location, variables m, n, p and q taking integer values in therange 1 to N such that the first and second sub-pixels or pixels arelocated diagonally with respect to the sub-pixel at ½^(N) unithorizontal and ½^(N) vertical location.

The communications terminal may comprise a video encoder. It maycomprise a video decoder. Preferably, it comprises a video codeccomprising a video encoder and a video decoder.

Preferably the communications terminal comprising a user interface, aprocessor and at least one of a transmitting block and a receivingblock, and a video coder according to at least one of the third andfourth aspects of the invention. Preferably the processor controls theoperation of the transmitting block and/or the receiving block and thevideo coder.

According to a sixth aspect of the invention, there is provided atelecommunications system comprising a communications terminal and anetwork, the telecommunications network and the communications terminalbeing connected by a communications link over which coded video can betransmitted, the communications terminal comprising a video coder forcoding for coding an image comprising pixels arranged in rows andcolumns and represented by values having a specified dynamic range, thepixels in the rows residing at unit horizontal locations and the pixelsin the columns residing at unit vertical locations, the video codercomprising an interpolator adapted to generate values for sub-pixels atfractional horizontal and vertical locations, the fractional horizontaland vertical locations being defined according to ½^(x), where x is apositive integer having a maximum value N, the interpolator beingadapted to:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate values for sub-pixels at ½^(N−1) unit horizontal and½^(N−1) unit vertical locations directly using a choice of a firstweighted sum of values for sub-pixels residing at ½^(N−1) unithorizontal and unit vertical locations and a second weighted sum ofvalues for sub-pixels residing at unit horizontal and ½^(N−1) unitvertical locations, the first and second weighted sums of values beingcalculated according to step (a); and

c) interpolate a value for a sub-pixel situated at a ½^(N) unithorizontal and ½^(N) unit vertical location by taking a weighted averageof the value of a first sub-pixel or pixel situated at a ½^(N−m) unithorizontal and ½^(N−n) unit vertical location and the value of a secondsub-pixel or pixel located at a ½^(N−p) unit horizontal and ½^(N−q) unitvertical location, variables m, n, p and q taking integer values in therange 1 to N such that the first and second sub-pixels or pixels arelocated diagonally with respect to the sub-pixel at ½^(N) unithorizontal and ½^(N) vertical location.

Preferably the telecommunications system is a mobile telecommunicationssystem comprising a mobile communications terminal and a wirelessnetwork, the connection between the mobile communications terminal andthe wireless network being formed by a radio link. Preferably thenetwork enables the communications terminal to communicate with othercommunications terminals connected to the network over communicationslinks between the other communications terminals and the network.

According to a seventh aspect of the invention, there is provided atelecommunications system comprising a communications terminal and anetwork, the telecommunications network and the communications terminalbeing connected by a communications link over which coded video can betransmitted, the network comprising a video coder for coding for codingan image comprising pixels arranged in rows and columns and representedby values having a specified dynamic range, the pixels in the rowsresiding at unit horizontal locations and the pixels in the columnsresiding at unit vertical locations, the video coder comprising aninterpolator adapted to generate values for sub-pixels at fractionalhorizontal and vertical locations, the fractional horizontal andvertical locations being defined according to ½^(x), where x is apositive integer having a maximum value N, the interpolator beingadapted to:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate values for sub-pixels at ½^(N−1) unit horizontal and½^(N−1) unit vertical locations directly using a choice of a firstweighted sum of values for sub-pixels residing at ½^(N−1) unithorizontal and unit vertical locations and a second weighted sum ofvalues for sub-pixels residing at unit horizontal and ½^(N−1) unitvertical locations, the first and second weighted sums of values beingcalculated according to step (a); and

c) interpolate a value for a sub-pixel situated at a ½^(N) unithorizontal and ½^(N) unit vertical location by taking a weighted averageof the value of a first sub-pixel or pixel situated at a ½^(N−m) unithorizontal and ½^(N−n) unit vertical location and the value of a secondsub-pixel or pixel located at a ½^(N−p) unit horizontal and ½^(N−q) unitvertical location, variables m, n, p and q taking integer values in therange 1 to N such that the first and second sub-pixels or pixels arelocated diagonally with respect to the sub-pixel at ½^(N) unithorizontal and ½^(N) vertical location.

According to an eighth aspect of the invention there is provided a videocoder for coding an image comprising pixels arranged in rows and columnsand represented by values having a specified dynamic range, the pixelsin the rows residing at unit horizontal locations and the pixels in thecolumns residing at unit vertical locations, the coder comprising aninterpolator adapted to generate values for sub-pixels at fractionalhorizontal and vertical locations, the resolution of the sub-pixelsbeing determined by a positive integer N, the interpolator being adaptedto:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate a value for a sub-pixel at a sub-pixel horizontal andsub-pixel vertical location is required directly using a choice of afirst weighted sum of values for sub-pixels located at a verticallocation corresponding to that of the sub-pixel being calculated and asecond weighted sum of values for sub-pixels located at a horizontallocation corresponding to that of the sub-pixel being calculated.

The interpolator may be further adapted to form the first weighted sumusing the values of sub-pixels residing at ½^(N−1) unit horizontal andunit vertical locations and to use the first weighted sum to interpolatea value for a sub-pixel at ½^(N−1) unit horizontal and ½^(N) unitvertical location.

The interpolator may be further adapted to form the second weighted sumusing the values of sub-pixels residing at unit horizontal and ½^(N−1)unit vertical locations and to use the second weighted sum tointerpolate a value for a sub-pixel at ½^(N) unit horizontal and ½^(N−1)unit vertical location.

The interpolator may be further adapted to interpolate values forsub-pixels at ½^(N) unit horizontal and ½^(N) unit vertical locations bytaking the average of at least one pair of a first pair of values of asub-pixel located at a ½^(N−1) unit horizontal and unit verticallocation, and a sub-pixel located at a unit horizontal and ½^(N−1) unitvertical location and a second pair of values of a pixel located at aunit horizontal and unit vertical location, and a sub-pixel located at a½^(N−1) unit horizontal and ½^(N−1) unit vertical location.

According to a ninth aspect of the invention there is provided acommunications terminal comprising a video coder for coding an imagecomprising pixels arranged in rows and columns and represented by valueshaving a specified dynamic range, the pixels in the rows residing atunit horizontal locations and the pixels in the columns residing at unitvertical locations, the coder comprising an interpolator adapted togenerate values for sub-pixels at fractional horizontal and verticallocations, the resolution of the sub-pixels being determined by apositive integer N, the interpolator being adapted to:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate a value for a sub-pixel at a sub-pixel horizontal andsub-pixel vertical location is required directly using a choice of afirst weighted sum of values for sub-pixels located at a verticallocation corresponding to that of the sub-pixel being calculated and asecond weighted sum of values for sub-pixels located at a horizontallocation corresponding to that of the sub-pixel being calculated.

According to a tenth aspect of the invention there is provided atelecommunications system comprising a communications terminal and anetwork, the telecommunications network and the communications terminalbeing connected by a communications link over which coded video can betransmitted, the communications terminal comprising a video coder forcoding an image comprising pixels arranged in rows and columns andrepresented by values having a specified dynamic range, the pixels inthe rows residing at unit horizontal locations and the pixels in thecolumns residing at unit vertical locations, the coder comprising aninterpolator adapted to generate values for sub-pixels at fractionalhorizontal and vertical locations, the resolution of the sub-pixelsbeing determined by a positive integer N, the interpolator being adaptedto:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate a value for a sub-pixel at a sub-pixel horizontal andsub-pixel vertical location is required directly using a choice of afirst weighted sum of values for sub-pixels located at a verticallocation corresponding to that of the sub-pixel being calculated and asecond weighted sum of values for sub-pixels located at a horizontallocation corresponding to that of the sub-pixel being calculated.

According to an eleventh aspect of the invention there is provided atelecommunications system comprising a communications terminal and anetwork, the telecommunications network and the communications terminalbeing connected by a communications link over which coded video can betransmitted, the network comprising a video coder for coding for codingan image comprising pixels arranged in rows and columns and representedby values having a specified dynamic range, the pixels in the rowsresiding at unit horizontal locations and the pixels in the columnsresiding at unit vertical locations, the coder comprising aninterpolator adapted to generate values for sub-pixels at fractionalhorizontal and vertical locations, the resolution of the sub-pixelsbeing determined by a positive integer N, the interpolator being adaptedto:

a) interpolate values for sub-pixels at ½^(N−1) unit horizontal and unitvertical locations, and unit horizontal and ½^(N−1) unit verticallocations directly using weighted sums of pixels residing at unithorizontal and unit vertical locations;

b) interpolate a value for a sub-pixel at a sub-pixel horizontal andsub-pixel vertical location is required directly using a choice of afirst weighted sum of values for sub-pixels located at a verticallocation corresponding to that of the sub-pixel being calculated and asecond weighted sum of values for sub-pixels located at a horizontallocation corresponding to that of the sub-pixel being calculated.

According to the twelfth aspect of the invention, the disclosedembodiments are directed to a method for sub-pixel value interpolationto determine values for sub-pixels situated within a rectangular boundedregion defined by four corner pixels with no intermediate pixels betweenthe corners, the pixels and sub-pixels being arranged in rows andcolumns, the pixel and sub-pixel locations being representablemathematically within the rectangular bounded region using theco-ordinate notation K/2^(N), L/2^(N), K and L being positive integershaving respective values between zero and 2^(N), N being a positiveinteger greater than one and representing a particular degree ofsub-pixel value interpolation, the method comprising:

-   -   (a) interpolating a sub-pixel value for a sub-pixel having        co-ordinates with odd values of both K and L, according to a        predetermined choice of a weighted average of the value of a        nearest-neighbouring pixel and the value of the sub-pixel        situated at co-ordinates ½, ½, and a weighted average of the        values of a pair of diagonally-opposed sub-pixels having        co-ordinates with even values of both K and L, including zero,        situated within a quadrant of the rectangular bounded region        defined by corner pixels having co-ordinates ½, ½ and the        nearest neighbouring pixel;    -   (b) interpolating sub-pixel values for sub-pixels having        co-ordinates with K equal to an even value and L equal zero and        sub-pixels having co-ordinates with K equal to zero and L equal        to an even value, used in the interpolation of the sub-pixels        having co-ordinates with odd values of both K and L, using        weighted sums of the values of pixels located in rows and        columns respectively; and    -   (c) interpolating sub-pixel values for sub-pixels having        co-ordinates with even values of both K and L, used in the        interpolation of sub-pixel values for the sub-pixels having        co-ordinates with odd values of both K and L, using a        predetermined choice of either a weighted sum of the values of        sub-pixels having co-ordinates with K equal to an even value and        L equal to zero and the values of sub-pixels having        corresponding co-ordinates in immediately adjacent rectangular        bounded regions, or a weighted sum of the values of sub-pixels        having co-ordinates with K equal to zero and L equal to an even        value and the values of sub-pixels having corresponding        co-ordinates in immediately adjacent bounded rectangular        regions.

According to the thirteenth aspect of the invention, the disclosedembodiments are directed to a method for quarter resolution sub-pixelvalue interpolation to determine values for sub-pixels situated within arectangular bounded region defined by four corner pixels with nointermediate pixels between the corners, the pixels and sub-pixels beingarranged in rows and columns, the pixel and sub-pixel locations beingrepresentable mathematically within the rectangular bounded region usingthe co-ordinate notation K/4, L/4, K and L being positive integershaving respective values between zero and 4, the method comprising:

-   -   (a) interpolating a sub-pixel value for a sub-pixel having        co-ordinates with both K and L equal to 1 or 3, according to a        predetermined choice of a weighted average of the value of a        nearest-neighbouring pixel and the value of the sub-pixel        situated at co-ordinates 2/4, 2/4, and a weighted average of the        values of a pair of diagonally-opposed sub-pixels having        co-ordinates with both K and L equal to zero, 2 or 4, situated        within a quadrant of the rectangular bounded region defined by        corner pixels having co-ordinates 2/4, 2/4 and the nearest        neighbouring pixel;    -   (b) interpolating sub-pixel values for sub-pixels having        co-ordinates with K equal to 2 and L equal zero and sub-pixels        having co-ordinates with K equal to zero and L equal to 2, used        in the interpolation of the sub-pixels having co-ordinates with        both K and L equal to 1 or 3, using weighted sums of the values        of pixels located in rows and columns respectively; and    -   (c) interpolating a sub-pixel value for the sub-pixel having        co-ordinates with both K and L equal to 2, used in the        interpolation of sub-pixel values for the sub-pixels having        co-ordinates with both K and L equal to 1 or 3, using a        predetermined choice of either a weighted sum of the values of        the sub-pixel having co-ordinates with K equal to 2 and L equal        to zero and the values of sub-pixels having corresponding        co-ordinates in immediately adjacent rectangular bounded        regions, or a weighted sum of the value of the sub-pixel having        co-ordinates with K equal to zero and L equal to 2 and the        values of sub-pixels having corresponding co-ordinates in        immediately adjacent rectangular bounded regions.

According to the fourteenth aspect of the invention, the disclosedembodiments are direct to a method for eighth resolution sub-pixel valueinterpolation to determine values for sub-pixels situated within arectangular bounded region defined by four corner pixels with nointermediate pixels between the corners, the pixels and sub-pixels beingarranged in rows and columns, the pixel and sub-pixel locations beingrepresentable mathematically with the rectangular bounded region usingthe co-ordinate notation K/8, L/8, K and L being positive integershaving respective values between zero and 8, the method comprising:

-   -   (a) interpolating a sub-pixel value for a sub-pixel having        co-ordinates with both K and L equal to 1, 3, 5 or 7, according        to a predetermined choice of a weighted average of the value of        a nearest-neighbouring pixel and the value of the sub-pixel        situated at co-ordinates 4/8, 4/8, and a weighted average of the        values of a pair of diagonally-opposed sub-pixels having        co-ordinates with both K and L equal to zero, 2, 4, 6 or 8,        situated within a quadrant of the rectangular bounded region        defined by corner pixels having co-ordinates 4/8, 4/8 and the        nearest neighbouring pixel;    -   (b) interpolating sub-pixel values for sub-pixels having        co-ordinates with K equal to 2, 4 or 6 and L equal zero and        sub-pixels having co-ordinates with K equal to zero and L equal        to 2, 4 or 6, used in the interpolation of the sub-pixels having        co-ordinates with both K and L equal to 1, 3, 5 or 7, using        weighted sums of the values of pixels located in rows and        columns respectively; and    -   (c) interpolating sub-pixel values for sub-pixels having        co-ordinates with both K and L equal to 2, 4 or 6, used in the        interpolation of sub-pixel values for the sub-pixels having        co-ordinates with both K and L equal to 1, 3, 5 or 7, using a        predetermined choice of either a weighted sum of the values of        sub-pixels having co-ordinates with K equal to 2, 4 or 6 and L        equal to zero and the values of sub-pixels having corresponding        co-ordinates in immediately adjacent rectangular bounded regions        or a weighted sum of the values of sub-pixels having        co-ordinates with K equal to zero and L equal to 2, 4 or 6 and        the values of sub-pixels having corresponding co-ordinates in        immediately adjacent rectangular bounded regions.

According to the fifteenth aspect of the invention, the disclosedembodiments are directed to an apparatus for sub-pixel valueinterpolation, the apparatus being operable to determine values forsub-pixels situated within a rectangular bounded region defined by fourcorner pixels with no intermediate pixels between the corners, thepixels and sub-pixels being arranged in rows and columns, the pixel andsub-pixel locations being representable mathematically within therectangular bounded region using the co-ordinate notation K/2^(N),L/2^(N), K and L being positive integers having respective valuesbetween zero and 2^(N), N being a positive integer greater than one andrepresenting a particular degree of sub-pixel value interpolation, theapparatus comprising:

-   -   (a) circuitry operable to interpolate a sub-pixel value for a        sub-pixel having co-ordinates with odd values of both K and L,        according to a predetermined choice of a weighted average of the        value of a nearest-neighbouring pixel and the value of the        sub-pixel situated at co-ordinates ½, ½, and a weighted average        of the values of a pair of diagonally-opposed sub-pixels having        co-ordinates with even values of both K and L, including zero,        situated within a quadrant of the rectangular bounded region        defined by corner pixels having co-ordinates ½, ½ and the        nearest neighbouring pixel;    -   (b) circuitry operable to interpolate sub-pixel values for        sub-pixels having co-ordinates with K equal to an even value and        L equal zero and sub-pixels having co-ordinates with K equal to        zero and L equal to an even value, used in the interpolation of        the sub-pixels having co-ordinates with odd values of both K and        L, using weighted sums of the values of pixels located in rows        and columns respectively; and    -   (c) circuitry operable to interpolate sub-pixel values for        sub-pixels having co-ordinates with even values of both K and L,        used in the interpolation of sub-pixel values for the sub-pixels        having co-ordinates with odd values of both K and L, using a        predetermined choice of either a weighted sum of the values of        sub-pixels having co-ordinates with K equal to an even value and        L equal to zero and the values of sub-pixels having        corresponding co-ordinates in immediately adjacent rectangular        bounded regions, or a weighted sum of the values of sub-pixels        having co-ordinates with K equal to zero and L equal to an even        value and the values of sub-pixels having corresponding        co-ordinates in immediately adjacent bounded rectangular        regions.

In one embodiment the method includes comprising using a first and asecond weight in the weighted average chosen to interpolate thesub-pixel value for a sub-pixel having co-ordinates with odd values ofboth K and L, wherein when the chosen weighted average is that involvingthe value of a nearest neighbouring pixel and the value of the sub-pixelsituated at co-ordinates ½, ½, selecting the relative magnitudes of thefirst and second weights to be inversely proportional to the respectivestraight-line diagonal distances of the nearest-neighbouring pixel andthe sub-pixel situated at co-ordinates ½, ½ to the sub-pixel havingco-ordinates with odd values of both K and L whose value is beinginterpolated.

In one embodiment the method includes comprising using a first and asecond weight in the weighted average chosen to interpolate thesub-pixel value for a sub-pixel having co-ordinates with odd values ofboth K and L, wherein when the chosen weighted average is that involvingthe values of a pair of diagonally-opposed sub-pixels havingco-ordinates with even values of both K and L, selecting the relativemagnitudes of the first and second weights to be inversely proportionalto the respective straight-line diagonal distances of the sub-pixelshaving co-ordinates with even values of both K and L to the sub-pixelhaving co-ordinates with odd values of both K and L whose value is beinginterpolated.

In one embodiment the method includes comprising using first and secondweights with equal values when the nearest-neighbouring pixel andsub-pixel situated at co-ordinates ½, ½ are equidistant from thesub-pixel having co-ordinates with odd values of both K and L whosevalue is being interpolated.

In one embodiment the method includes comprising using first and secondweights with equal values when the sub-pixels having co-ordinates witheven values of both K and L are equidistant from the sub-pixel havingco-ordinates with odd values of both K and L whose value is beinginterpolated.

In one embodiment the method includes interpolating sub-pixel values forsub-pixels having co-ordinates with even values of both K and L, used ininterpolation of sub-pixel values for the sub-pixels having co-ordinateswith odd values of both K and L, using a weighted sum of the values ofsub-pixels having co-ordinates with K equal to an even value and L equalto zero and the values of sub-pixels having corresponding co-ordinatesin immediately adjacent rectangular bounded regions when the value of asub-pixel having co-ordinates with K equal to an even value and L equalto an odd value is also required.

In one embodiment the method includes comprising interpolating sub-pixelvalues for sub-pixels having co-ordinates with even values of both K andL, used in interpolation of sub-pixel values for the sub-pixels havingco-ordinates with odd values of both K and L, using a weighted sum ofthe values of sub-pixels having co-ordinates with K equal to zero and Lequal to an even value and the values of sub-pixels having correspondingco-ordinates in immediately adjacent rectangular bounded regions whenthe value of a sub-pixel having co-ordinates with K equal to an oddvalue and L equal to an even value is also required.

In one embodiment the method includes comprising interpolating sub-pixelvalues for sub-pixels having co-ordinates with K equal to an odd value Oand L equal to an even value E, not including zero or 2^(N), by takingan average of the value of a first sub-pixel having co-ordinates with Kequal to the even value O−1 and L equal to E and a second sub-pixelhaving co-ordinates with K equal to the even value O+1 and L equal to E.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to 1and L equal to zero, by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to 2 and L equal to zero.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to2^(N)−1 and L equal to zero, by taking an average of the value of apixel having co-ordinates with K equal to 2^(N) and L equal to zero, anda sub-pixel having co-ordinates with K equal to 2^(N)−2 and L equal tozero.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to 1and L equal to 2^(N), by taking an average of the value of a pixelhaving co-ordinates with K equal to zero and L equal to 2^(N), and asub-pixel having co-ordinates with K equal to 2 and L equal to 2^(N).

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to2^(N)−1 and L equal to 2^(N), by taking an average of the value of apixel having co-ordinates with K equal to 2^(N) and L equal to 2^(N),and a sub-pixel having co-ordinates with K equal to 2^(N)−2 and L equalto 2^(N).

In one embodiment the method includes comprising interpolating sub-pixelvalues for sub-pixels having co-ordinates with K equal to an even valueE, not including zero or 2^(N), and L equal to an odd value O, by takingan average of the value of a first sub-pixel having co-ordinates with Kequal to E and L equal to the even value O−1 and a second sub-pixelhaving co-ordinates with K equal to E and L equal to the even value O+1.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to zeroand L equal to 1, by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to zero and L equal to 2.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to2^(N) and L equal to 1, by taking an average of the value of a pixelhaving co-ordinates with K equal to 2^(N) and L equal to zero, and asub-pixel having co-ordinates with K equal to zero and L equal to 2.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to 0and L equal to 2^(N)−1, by taking an average of the value of a pixelhaving co-ordinates with K equal to zero and L equal to 2^(N), and asub-pixel having co-ordinates with K equal to zero and L equal to2^(N)−2.

In one embodiment the method includes comprising interpolating asub-pixel value for a sub-pixel having co-ordinates with K equal to2^(N) and L equal to 2^(N)−1, by taking an average of the value of apixel having co-ordinates with K equal to 2^(N) and L equal to 2^(N),and a sub-pixel having co-ordinates with K equal to 2^(N) and L equal to2^(N)−2.

In one embodiment the method includes comprising interpolating asub-pixel value for the sub-pixel having co-ordinates with K equal to2^(N)−1 and L equal to 2^(N)−1, by taking a weighted average of thevalues of the four corner pixels that define the rectangular boundedregion.

In one embodiment the method includes comprising selecting a value for Nfrom a list consisting of the values 2, 3, and 4.

In one embodiment the apparatus is operable to use a first and a secondweight in the weighted average chosen to interpolate the sub-pixel valuefor a sub-pixel having co-ordinates with odd values of both K and L,wherein when the chosen weighted average is that involving the value ofa nearest neighbouring pixel and the value of the sub-pixel situated atco-ordinates ½, ½, the apparatus is operable to select the relativemagnitudes of the first and second weights to be inversely proportionalto the respective straight-line diagonal distances of thenearest-neighbouring pixel and the sub-pixel situated at co-ordinates ½,½ to the sub-pixel having co-ordinates with odd values of both K and Lwhose value is being interpolated.

In one embodiment the apparatus is operable to use a first and a secondweight in the weighted average chosen to interpolate the sub-pixel valuefor a sub-pixel having co-ordinates with odd values of both K and L,wherein when the chosen weighted average is that involving the values ofa pair of diagonally-opposed sub-pixels having co-ordinates with evenvalues of both K and L, the apparatus is operable to select the relativemagnitudes of the first and second weights to be inversely proportionalto the respective straight-line diagonal distances of the sub-pixelshaving co-ordinates with even values of both K and L to the sub-pixelhaving co-ordinates with odd values of both K and L whose value is beinginterpolated.

In one embodiment the apparatus is operable to use first and secondweights with equal values when the nearest-neighbouring pixel andsub-pixel situated at co-ordinates ½, ½ are equidistant from thesub-pixel having co-ordinates with odd values of both K and L whosevalue is being interpolated.

In one embodiment the apparatus is operable to use first and secondweights with equal values when the sub-pixels having co-ordinates witheven values of both K and L are equidistant from the sub-pixel havingco-ordinates with odd values of both K and L whose value is beinginterpolated.

In one embodiment the apparatus is operable to interpolate sub-pixelvalues for sub-pixels having co-ordinates with even values of both K andL, used in interpolation of sub-pixel values for the sub-pixels havingco-ordinates with odd values of both K and L, using a weighted sum ofthe values of sub-pixels having co-ordinates with K equal to an evenvalue and L equal to zero and the values of sub-pixels havingcorresponding co-ordinates in immediately adjacent rectangular boundedregions when the value of a sub-pixel having co-ordinates with K equalto an even value and L equal to an odd value is also required.

In one embodiment the apparatus is operable to interpolate sub-pixelvalues for sub-pixels having co-ordinates with even values of both K andL, used in interpolation of sub-pixel values for the sub-pixels havingco-ordinates with odd values of both K and L, using a weighted sum ofthe values of sub-pixels having co-ordinates with K equal to zero and Lequal to an even value and the values of sub-pixels having correspondingco-ordinates in immediately adjacent rectangular bounded regions whenthe value of a sub-pixel having co-ordinates with K equal to an oddvalue and L equal to an even value is also required.

In one embodiment the apparatus is operable to interpolate sub-pixelvalues for sub-pixels having co-ordinates with K equal to an odd value Oand L equal to an even value E, not including zero or 2^(N), by takingan average of the value of a first sub-pixel having co-ordinates with Kequal to the even value O−1 and L equal to E and a second sub-pixelhaving co-ordinates with K equal to the even value O+1 and L equal to E.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 1 and L equalto zero, by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to 2 and L equal to zero.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 2^(N)−1 and Lequal to zero, by taking an average of the value of a pixel havingco-ordinates with K equal to 2^(N) and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to 2^(N)−2 and L equal to zero.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 1 and L equalto 2^(N), by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to 2^(N), and a sub-pixelhaving co-ordinates with K equal to 2 and L equal to 2^(N).

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 2^(N)−1 and Lequal to 2^(N), by taking an average of the value of a pixel havingco-ordinates with K equal to 2^(N) and L equal to 2^(N), and a sub-pixelhaving co-ordinates with K equal to 2^(N)−2 and L equal to 2^(N)

In one embodiment the apparatus is operable to interpolate sub-pixelvalues for sub-pixels having co-ordinates with K equal to an even valueE, not including zero or 2^(N), and L equal to an odd value O, by takingan average of the value of a first sub-pixel having co-ordinates with Kequal to E and L equal to the even value O−1 and a second sub-pixelhaving co-ordinates with K equal to E and L equal to the even value O+1.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to zero and Lequal to 1, by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to zero and L equal to 2.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 2^(N) and Lequal to 1, by taking an average of the value of a pixel havingco-ordinates with K equal to 2^(N) and L equal to zero, and a sub-pixelhaving co-ordinates with K equal to zero and L equal to 2.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 0 and L equalto 2^(N)−1, by taking an average of the value of a pixel havingco-ordinates with K equal to zero and L equal to 2^(N), and a sub-pixelhaving co-ordinates with K equal to zero and L equal to 2^(N)−2.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for a sub-pixel having co-ordinates with K equal to 2^(N) and Lequal to 2^(N)−1, by taking an average of the value of a pixel havingco-ordinates with K equal to 2^(N) and L equal to 2^(N), and a sub-pixelhaving co-ordinates with K equal to 2^(N) and L equal to 2^(N)−2.

In one embodiment the apparatus is operable to interpolate a sub-pixelvalue for the sub-pixel having co-ordinates with K equal to 2^(N)−1 andL equal to 2^(N)−1, by taking a weighted average of the values of thefour corner pixels that define the rectangular bounded region.

In one embodiment the apparatus includes N is set equal to 2.

In one embodiment the apparatus includes N is set equal to 3.

In one embodiment the apparatus includes a video encoder comprising anapparatus for sub-pixel value interpolation.

In one embodiment the apparatus includes a still image encodercomprising an apparatus for sub-pixel value interpolation.

In one embodiment the apparatus includes a video decoder comprising anapparatus for sub-pixel value interpolation.

In one embodiment the apparatus includes a still image decodercomprising an apparatus for sub-pixel value interpolation.

In one embodiment the apparatus includes a codec comprising a videoencoder and a video decoder.

In one embodiment the apparatus includes a codec comprising a stillimage encoder and a still image decoder.

In one embodiment the apparatus includes a communications terminalcomprising a video encoder.

In one embodiment the apparatus includes a communications terminalcomprising a still image encoder.

In one embodiment the apparatus includes a communications terminalcomprising a video decoder.

In one embodiment the apparatus includes a communications terminalcomprising a still image decoder.

BRIEF DESCRIPTION OF THE FIGURES

An embodiment of the invention will now be described by way of exampleonly with reference to the accompanying drawings in which:

FIG. 1 shows a video encoder according to the prior art;

FIG. 2 shows a video decoder according to the prior art;

FIG. 3 shows the types of frames used in video encoding;

FIGS. 4 a, 4 b, and 4 c show steps in block-matching;

FIG. 5 illustrates the process of motion estimation to sub-pixelresolution;

FIG. 6 shows a terminal device comprising video encoding and decodingequipment in which the method of the invention may be implemented;

FIG. 7 shows a video encoder according an embodiment of the presentinvention;

FIG. 8 shows a video decoder according to an embodiment of the presentinvention;

FIGS. 9 and 10 have not been used and any such figures should bedisregarded;

FIG. 11 shows a schematic diagram of a mobile telecommunications networkaccording to an embodiment of the present invention;

FIG. 12 a shows a notation for describing pixel and sub-pixel locationsspecific to TML5;

FIG. 12 b shows interpolation of a half resolution sub-pixels;

FIG. 12 c shows interpolation of a half resolution sub-pixels;

FIG. 13 a shows a notation for describing pixel and sub-pixel locationsspecific to TML6;

FIG. 13 b shows interpolation of a half resolution sub-pixels;

FIG. 13 c shows interpolation of a half resolution sub-pixels;

FIG. 14 shows a notation for describing pixel and sub-pixel locationsspecific to the invention;

FIG. 14 b shows interpolation of a half resolution sub-pixels accordingto the invention;

FIG. 14 c shows interpolation of a half resolution sub-pixels accordingto the invention;

FIG. 15 shows possible choices of diagonal interpolation for sub-pixels;

FIG. 16 shows the half resolution sub-pixel values required to calculateother half resolution sub-pixel values;

FIG. 17 a shows the half resolution sub-pixel values that must becalculated in order to interpolate values for quarter resolutionsub-pixels in an image block using the interpolation method of TML5;

FIG. 17 b shows the half resolution sub-pixel values that must becalculated in order to interpolate values for quarter resolutionsub-pixels in an image block using the interpolation method according tothe invention;

FIG. 18 a shows the numbers of half resolution sub-pixels that must becalculated in order to obtain values for quarter resolution sub-pixelswithin an image block using the sub-pixel value interpolation methodaccording to TML5;

FIG. 18 b shows the numbers of half resolution sub-pixels that must becalculated in order to obtain values for quarter resolution sub-pixelswithin an image block using the sub-pixel value interpolation methodaccording to the invention;

FIG. 19 shows a numbering scheme for each of the 15 sub-pixel positions;

FIG. 20 shows nomenclature used to describe pixels, half resolutionsub-pixels, quarter resolution sub-pixels and eighth resolutionsub-pixels

FIG. 21 a shows the diagonal direction to be used in the interpolationof each eighth resolution sub-pixel in an embodiment of the invention;

FIG. 21 b shows the diagonal direction to be used in the interpolationof each eighth resolution sub-pixel in another embodiment of theinvention; and

FIG. 22 shows nomenclature used to describe eighth resolution sub-pixelswithin an image.

DETAILED DESCRIPTION

FIGS. 1 to 5, 12 a, 12 b, 12 c, 13 a, 13 b, and 13 c have been describedin the foregoing.

FIG. 6 presents a terminal device comprising video encoding and decodingequipment which may be adapted to operate in accordance with the presentinvention. More precisely, the figure illustrates a multimedia terminal60 implemented according to ITU-T recommendation H.324. The terminal canbe regarded as a multimedia transceiver device. It includes elementsthat capture, encode and multiplex multimedia data streams fortransmission via a communications network, as well as elements thatreceive, de-multiplex, decode and display received multimedia content.ITU-T recommendation H.324 defines the overall operation of the terminaland refers to other recommendations that govern the operation of itsvarious constituent parts. This kind of multimedia terminal can be usedin real-time applications such as conversational videotelephony, or nonreal-time applications such as the retrieval/streaming of video clips,for example from a multimedia content server in the Internet.

In the context of the present invention, it should be appreciated thatthe H.324 terminal shown in FIG. 6 is only one of a number ofalternative multimedia terminal implementations suited to application ofthe inventive method. It should also be noted that a number ofalternatives exist relating to the location and implementation of theterminal equipment. As illustrated in FIG. 6, the multimedia terminalmay be located in communications equipment connected to a fixed linetelephone network such as an analogue PSTN (Public Switched TelephoneNetwork). In this case the multimedia terminal is equipped with a modem71, compliant with ITU-T recommendations V.8, V.34 and optionallyV.8bis. Alternatively, the multimedia terminal may be connected to anexternal modem. The modem enables conversion of the multiplexed digitaldata and control signals produced by the multimedia terminal into ananalogue form suitable for transmission over the PSTN. It furtherenables the multimedia terminal to receive data and control signals inanalogue form from the PSTN and to convert them into a digital datastream that can be demultiplexed and processed in an appropriate mannerby the terminal.

An H.324 multimedia terminal may also be implemented in such a way thatit can be connected directly to a digital fixed line network, such as anISDN (Integrated Services Digital Network). In this case the modem 71 isreplaced with an ISDN user-network interface. In FIG. 6, this ISDNuser-network interface is represented by alternative block 72.

H.324 multimedia terminals may also be adapted for use in mobilecommunication applications. If used with a wireless communication link,the modem 71 can be replaced with any appropriate wireless interface, asrepresented by alternative block 73 in FIG. 6. For example, an H.324/Mmultimedia terminal can include a radio transceiver enabling connectionto the current 2^(nd) generation GSM mobile telephone network, or theproposed 3^(rd) generation UMTS (Universal Mobile Telephone System).

It should be noted that in multimedia terminals designed for two-waycommunication, that is for transmission and reception of video data, itis advantageous to provide both a video encoder and video decoderimplemented according to the present invention. Such an encoder anddecoder pair is often implemented as a single combined functional unit,referred to as a ‘codec’.

Because a video encoder according to the invention performs motioncompensated video encoding to sub-pixel resolution using a specificinterpolation scheme and a particular combination of before-hand andon-demand sub-pixel value interpolation, it is generally necessary for avideo decoder of a receiving terminal to be implemented in a mannercompatible with the encoder of the transmitting terminal which formedthe compressed video data stream. Failure to ensure this compatibilitymay have an adverse effect on the quality of the motion compensation andthe accuracy of reconstructed video frames.

A typical H.324 multimedia terminal will now be described in furtherdetail with reference to FIG. 6.

The multimedia terminal 60 includes a variety of elements referred to as‘terminal equipment’. This includes video, audio and telematic devices,denoted generically by reference numbers 61, 62 and 63, respectively.The video equipment 61 may include, for example, a video camera forcapturing video images, a monitor for displaying received video contentand optional video processing equipment. The audio equipment 62typically includes a microphone, for example for capturing spokenmessages, and a loudspeaker for reproducing received audio content. Theaudio equipment may also include additional audio processing units. Thetelematic equipment 63, may include a data terminal, keyboard,electronic whiteboard or a still image transceiver, such as a fax unit.

The video equipment 61 is coupled to a video codec 65. The video codec65 comprises a video encoder and a corresponding video decoder bothimplemented according to the invention. Such an encoder and a decoderwill be described in the following. The video codec 65 is responsiblefor encoding captured video data in an appropriate form for furthertransmission over a communications link and decoding compressed videocontent received from the communications network. In the exampleillustrated in FIG. 6, the video codec is implemented according to ITU-Trecommendation H.263, with appropriate modifications to implement thesub-pixel value interpolation method according to the invention in boththe encoder and the decoder of the video codec.

Similarly, the terminal's audio equipment is coupled to an audio codec,denoted in FIG. 6 by reference number 66. Like the video codec, theaudio codec comprises an encoder/decoder pair. It converts audio datacaptured by the terminal's audio equipment into a form suitable fortransmission over the communications link and transforms encoded audiodata received from the network back into a form suitable forreproduction, for example on the terminal's loudspeaker. The output ofthe audio codec is passed to a delay block 67. This compensates for thedelays introduced by the video coding process and thus ensuressynchronisation of audio and video content.

The system control block 64 of the multimedia terminal controlsend-to-network signalling using an appropriate control protocol(signalling block 68) to establish a common mode of operation between atransmitting and a receiving terminal. The signalling block 68 exchangesinformation about the encoding and decoding capabilities of thetransmitting and receiving terminals and can be used to enable thevarious coding modes of the video encoder. The system control block 64also controls the use of data encryption. Information regarding the typeof encryption to be used in data transmission is passed from encryptionblock 69 to the multiplexer/de-multiplexer (MUX/DMUX unit) 70.

During data transmission from the multimedia terminal, the MUX/DMUX unit70 combines encoded and synchronised video and audio streams with datainput from the telematic equipment 63 and possible control data, to forma single bit-stream. Information concerning the type of data encryption(if any) to be applied to the bit-stream, provided by encryption block69, is used to select an encryption mode. Correspondingly, when amultiplexed and possibly encrypted multimedia bit-stream is beingreceived, MUX/DMUX unit 70 is responsible for decrypting the bit-stream,dividing it into its constituent multimedia components and passing thosecomponents to the appropriate codec(s) and/or terminal equipment fordecoding and reproduction.

It should be noted that the functional elements of the multimediaterminal, video encoder, decoder and video codec according to theinvention can be implemented as software or dedicated hardware, or acombination of the two. The video encoding and decoding methodsaccording to the invention are particularly suited for implementation inthe form of a computer program comprising machine-readable instructionsfor performing the functional steps of the invention. As such, theencoder and decoder according to the invention may be implemented assoftware code stored on a storage medium and executed in a computer,such as a personal desktop computer, in order to provide that computerwith video encoding and/or decoding functionality.

If the multimedia terminal 60 is a mobile terminal, that is if it isequipped with a radio transceiver 73, it will be understood by thoseskilled in the art that it may also comprise additional elements. In oneembodiment it comprises a user interface having a display and akeyboard, which enables operation of the multimedia terminal 60 by auser, together with necessary functional blocks including a centralprocessing unit, such as a microprocessor, which controls the blocksresponsible for different functions of the multimedia terminal, a randomaccess memory RAM, a read only memory ROM, and a digital camera. Themicroprocessor's operating instructions, that is program codecorresponding to the basic functions of the multimedia terminal 60, isstored in the read-only memory ROM and can be executed as required bythe microprocessor, for example under control of the user. In accordancewith the program code, the microprocessor uses the radio transceiver 73to form a connection with a mobile communication network, enabling themultimedia terminal 60 to transmit information to and receiveinformation from the mobile communication network over a radio path.

The microprocessor monitors the state of the user interface and controlsthe digital camera. In response to a user command, the microprocessorinstructs the camera to record digital images into the RAM. Once animage is captured, or alternatively during the capturing process, themicroprocessor segments the image into image segments (for examplemacroblocks) and uses the encoder to perform motion compensated encodingfor the segments in order to generate a compressed image sequence asexplained in the foregoing description. A user may command themultimedia terminal 60 to display the captured images on its display orto send the compressed image sequence using the radio transceiver 73 toanother multimedia terminal, a video telephone connected to a fixed linenetwork (PSTN) or some other telecommunications device. In a preferredembodiment, transmission of image data is started as soon as the firstsegment is encoded so that the recipient can start a correspondingdecoding process with a minimum delay.

FIG. 11 is a schematic diagram of a mobile telecommunications networkaccording to an embodiment of the invention. Multimedia terminals MS arein communication with base stations BTS by means of a radio link. Thebase stations BTS are further connected, through a so-called Abisinterface, to a base station controller BSC, which controls and managesseveral base stations.

The entity formed by a number of base stations BTS (typically, by a fewtens of base stations) and a single base station controller BSC,controlling the base stations, is called a base station subsystem BSS.Particularly, the base station controller BSC manages radiocommunication channels and handovers. The base station controller BSC isalso connected, through a so-called A interface, to a mobile servicesswitching centre MSC, which co-ordinates the formation of connections toand from mobile stations. A further connection is made, through themobile service switching centre MSC, to outside the mobilecommunications network. Outside the mobile communications network theremay further reside other network(s) connected to the mobilecommunications network by gateway(s) GTW, for example the Internet or aPublic Switched Telephone Network (PSTN). In such an external network,or within the telecommunications network, there may be located videodecoding or encoding stations, such as computers PC. In an embodiment ofthe invention, the mobile telecommunications network comprises a videoserver VSRVR to provide video data to a MS subscribing to such aservice. The video data is compressed using the motion compensated videocompression method as described in the foregoing. The video server mayfunction as a gateway to an online video source or it may comprisepreviously recorded video clips. Typical videotelephony applications mayinvolve, for example, two mobile stations or one mobile station MS and avideotelephone connected to the PSTN, a PC connected to the Internet ora H.261 compatible terminal connected either to the Internet or to thePSTN.

FIG. 7 shows a video encoder 700 according to an embodiment theinvention. FIG. 8 shows a video decoder 800 according to an embodimentthe invention.

The encoder 700 comprises an input 701 for receiving a video signal froma camera or other video source (not shown). It further comprises a DCTtransformer 705, a quantiser 706, an inverse quantiser 709, an inverseDCT transformer 710, combiners 712 and 716, a before-hand sub-pixelinterpolation block 730, a frame store 740 and an on-demand sub-pixelinterpolation block 750, implemented in combination with motionestimation block 760. The encoder also comprises a motion field codingblock 770 and a motion compensated prediction block 780. Switches 702and 714 are operated co-operatively by a control manager 720 to switchthe encoder between an INTRA-mode of video encoding and an INTER-mode ofvideo encoding. The encoder 700 also comprises a multiplexer unit(MUX/DMUX) 790 to form a single bit-stream from the various types ofinformation produced by the encoder 700 for further transmission to aremote receiving terminal, or for example for storage on a mass storagemedium such as a computer hard drive (not shown).

It should be noted that the presence and implementations of before-handsub-pixel interpolation block 730 and on-demand sub-pixel valueinterpolation block 750 in the encoder architecture depend on the way inwhich the sub-pixel interpolation method according to the invention isapplied. In embodiments of the invention in which before-hand sub-pixelvalue interpolation is not performed, encoder 700 does not comprisebefore-hand sub-pixel value interpolation block 730. In otherembodiments of the invention, only before-hand sub-pixel interpolationis performed and thus the encoder does not include on-demand sub-pixelvalue interpolation block 750. In embodiments in which both before-handand on-demand sub-pixel value interpolation are performed, both blocks730 and 750 are present in the encoder 700.

Operation of the encoder 700 according to the invention will now bedescribed in detail. In the description, it will be assumed that eachframe of uncompressed video, received from the video source at the input701, is received and processed on a macroblock-by-macroblock basis,preferably in raster-scan order. It will further be assumed that whenthe encoding of a new video sequence starts, the first frame of thesequence is encoded in INTRA-mode. Subsequently, the encoder isprogrammed to code each frame in INTER-format, unless one of thefollowing conditions is met: 1) it is judged that the current framebeing coded is so dissimilar from the reference frame used in itsprediction that excessive prediction error information is produced; 2) apredefined INTRA frame repetition interval has expired; or 3) feedbackis received from a receiving terminal indicating a request for a frameto be coded in INTRA format.

The occurrence of condition 1) is detected by monitoring the output ofthe combiner 716. The combiner 716 forms a difference between thecurrent macroblock of the frame being coded and its prediction, producedin the motion compensated prediction block 780. If a measure of thisdifference (for example a sum of absolute differences of pixel values)exceeds a predetermined threshold, the combiner 716 informs the controlmanager 720 via a control line 717 and the control manager 720 operatesthe switches 702 and 714 so as to switch the encoder 700 into INTRAcoding mode. Occurrence of condition 2) is monitored by means of a timeror frame counter implemented in the control manager 720, in such a waythat if the timer expires, or the frame counter reaches a predeterminednumber of frames, the control manager 720 operates the switches 702 and714 to switch the encoder into INTRA coding mode. Condition 3) istriggered if the control manager 720 receives a feedback signal from,for example, a receiving terminal, via control line 718 indicating thatan INTRA frame refresh is required by the receiving terminal. Such acondition might arise, for example, if a previously transmitted framewere badly corrupted by interference during its transmission, renderingit impossible to decode at the receiver. In this situation, the receiverwould issue a request for the next frame to be encoded in INTRA format,thus re-initializing the coding sequence.

It will further be assumed that the encoder and decoder are implementedin such a way as to allow the determination of motion vectors with aspatial resolution of up to quarter-pixel resolution. As will be seen inthe following, finer levels of resolution are also possible.

Operation of the encoder 700 in INTRA coding mode will now be described.In INTRA-mode, the control manager 720 operates the switch 702 to acceptvideo input from input line 719. The video signal input is receivedmacroblock by macroblock from input 701 via the input line 719 and eachmacroblock of original image pixels is transformed into DCT coefficientsby the DCT transformer 705. The DCT coefficients are then passed to thequantiser 706, where they are quantised using a quantisation parameterQP. Selection of the quantisation parameter QP is controlled by thecontrol manager 720 via control line 722. Each DCT transformed andquantised macroblock that makes up the INTRA coded image information 723of the frame is passed from the quantiser 706 to the MUX/DMUX 790. TheMUX/DMUX 790 combines the INTRA coded image information with possiblecontrol information (for example header data, quantisation parameterinformation, error correction data etc.) to form a single bit-stream ofcoded image information 725. Variable length coding (VLC) is used toreduce redundancy of the compressed video bit-stream, as is known tothose skilled in the art.

A locally decoded picture is formed in the encoder 700 by passing thedata output by the quantiser 706 through inverse quantiser 709 andapplying an inverse DCT transform 710 to the inverse-quantised data. Theresulting data is then input to the combiner 712. In INTRA mode, switch714 is set so that the input to the combiner 712 via the switch 714 isset to zero. In this way, the operation performed by the combiner 712 isequivalent to passing the decoded image data formed by the inversequantiser 709 and the inverse DCT transform 710 unaltered.

In embodiments of the invention in which before-hand sub-pixel valueinterpolation is performed, the output from combiner 712 is applied tobefore-hand sub-pixel interpolation block 730. The input to thebefore-hand sub-pixel value interpolation block 730 takes the form ofdecoded image blocks. In the before-hand sub-pixel value interpolationblock 730, each decoded macroblock is subjected to sub-pixelinterpolation in such a way that a predetermined sub-set of sub-pixelresolution sub-pixel values is calculated according to the interpolationmethod of the invention and is stored together with the decoded pixelvalues in frame store 740.

In embodiments in which before-hand sub-pixel interpolation is notperformed, before-hand sub-pixel interpolation block is not present inthe encoder architecture and the output from combiner 712, comprisingdecoded image blocks, is applied directly to frame store 740.

As subsequent macroblocks of the current frame are received and undergothe previously described coding and decoding steps in blocks 705, 706,709, 710, 712, a decoded version of the INTRA frame is built up in theframe store 740. When the last macroblock of the current frame has beenINTRA coded and subsequently decoded, the frame store 740 contains acompletely decoded frame, available for use as a prediction referenceframe in coding a subsequently received video frame in INTER format. Inembodiments of the invention in which before-hand sub-pixel valueinterpolation is performed, the reference frame held in frame store 740is at least partially interpolated to sub-pixel resolution.

Operation of the encoder 700 in INTER coding mode will now be described.In INTER coding mode, the control manager 720 operates switch 702 toreceive its input from line 721, which comprises the output of thecombiner 716. The combiner 716 forms prediction error informationrepresenting the difference between the current macroblock of the framebeing coded and its prediction, produced in the motion compensatedprediction block 780. The prediction error information is DCTtransformed in block 705 and quantised in block 706 to form a macroblockof DCT transformed and quantised prediction error information. Eachmacroblock of DCT transformed and quantised prediction error informationis passed from the quantiser 706 to the MUX/DMUX unit 790. The MUX/DMUXunit 790 combines the prediction error information 723 with motioncoefficients 724 (described in the following) and control information(for example header data, quantisation parameter information, errorcorrection data etc.) to form a single bit-stream of coded imageinformation, 725.

Locally decoded prediction error information for the each macroblock ofthe INTER coded frame is then formed in the encoder 700 by passing theencoded prediction error information 723 output by the quantiser 706through the inverse quantiser 709 and applying an inverse DCT transformin block 710. The resulting locally decoded macroblock of predictionerror information is then input to combiner 712. In INTER-mode, switch714 is set so that the combiner 712 also receives motion predictedmacroblocks for the current INTER frame, produced in the motioncompensated prediction block 780. The combiner 712 combines these twopieces of information to produce reconstructed image blocks for thecurrent INTER frame.

As described above when considering INTRA coded frames, in embodimentsof the invention in which before-hand sub-pixel value interpolation isperformed, the output from combiner 712 is applied to the before-handsub-pixel interpolation block 730. Thus, the input to the before-handsub-pixel value interpolation block 730 in INTER coding mode also takesthe form of decoded image blocks. In the before-hand sub-pixel valueinterpolation block 730, each decoded macroblock is subjected tosub-pixel interpolation in such a way that a predetermined sub-set ofsub-pixel values is calculated according to the interpolation method ofthe invention and is stored together with the decoded pixel values inframe store 740. In embodiments in which before-hand sub-pixelinterpolation is not performed, before-hand sub-pixel interpolationblock is not present in the encoder architecture and the output fromcombiner 712, comprising decoded image blocks, is applied directly toframe store 740.

As subsequent macroblocks of the video signal are received from thevideo source and undergo the previously described coding and decodingsteps in blocks 705, 706, 709, 710, 712, a decoded version of the INTERframe is built up in the frame store 740. When the last macroblock ofthe frame has been INTER coded and subsequently decoded, the frame store740 contains a completely decoded frame, available for use as aprediction reference frame in encoding a subsequently received videoframe in INTER format. In embodiments of the invention in whichbefore-hand sub-pixel value interpolation is performed, the referenceframe held in frame store 740 is at least partially interpolated tosub-pixel resolution.

Formation of a prediction for a macroblock of the current frame will nowbe described.

Any frame encoded in INTER format requires a reference frame for motioncompensated prediction. This means, inter alia, that when encoding avideo sequence, the first frame to be encoded, whether it is the firstframe in the sequence, or some other frame, must be encoded in INTRAformat. This, in turn, means that when the video encoder 700 is switchedinto INTER coding mode by control manager 720, a complete referenceframe, formed by locally decoding a previously encoded frame, is alreadyavailable in the frame store 740 of the encoder. In general, thereference frame is formed by locally decoding either an INTRA codedframe or an INTER coded frame.

The first step in forming a prediction for a macroblock of the currentframe is performed by motion estimation block 760. The motion estimationblock 760 receives the current macroblock of the frame being coded vialine 727 and performs a block matching operation in order to identify aregion in the reference frame which corresponds substantially with thecurrent macroblock. According to the invention, the block-matchingprocess is performed to sub-pixel resolution in a manner that depends onthe implementation of the encoder 700 and the degree of before-handsub-pixel interpolation performed. However, the basic principle behindthe block-matching process is similar in all cases. Specifically, motionestimation block 760 performs block-matching by calculating differencevalues (e.g. sums of absolute differences) representing the differencein pixel values between the macroblock of the current frame underexamination and candidate best-matching regions of pixels/sub-pixels inthe reference frame. A difference value is produced for all possibleoffsets (e.g. quarter- or one eighth sub-pixel precision x, ydisplacements) between the macroblock of the current frame and candidatetest region within a predefined search region of the reference frame andmotion estimation block 760 determines the smallest calculateddifference value. The offset between the macroblock in the current frameand the candidate test region of pixel values/sub-pixel values in thereference frame that yields the smallest difference value defines themotion vector for the macroblock in question. In certain embodiments ofthe invention, an initial estimate for the motion vector having unitpixel precision is first determined and then refined to a finer level ofsub-pixel precision, as described in the foregoing.

In embodiments of the encoder in which before-hand sub-pixel valueinterpolation is not performed, all sub-pixel values required in theblock matching process are calculated in on-demand sub-pixel valueinterpolation block 750. Motion estimation block 760 controls on-demandsub-pixel value interpolation block 750 to calculate each sub-pixelvalue needed in the block-matching process in an on-demand fashion, asand when it is required. In this case, motion estimation block 760 maybe implemented so as to perform block-matching as a one-step process, inwhich case a motion vector with the desired sub-pixel resolution issought directly, or it may be implemented so as to performblock-matching as a two step process. If the two-step process isadopted, the first step may comprise a search for e.g. a full orhalf-pixel resolution motion vector and the second step is performed inorder to refine the motion vector to the desired sub-pixel resolution.As block matching is an exhaustive process, in which blocks of n×mpixels in the current frame are compared one-by-one with blocks of n×mpixels or sub-pixels in the interpolated reference frame, it should beappreciated that a sub-pixel calculated in an on-demand fashion by theon-demand pixel interpolation block 750 may need to be calculatedmultiple times as successive difference values are determined. In avideo encoder, this approach is not the most efficient possible in termsof computational complexity/burden.

In embodiments of the encoder which use only before-hand sub-pixel valueinterpolation, block-matching may be performed as a one step process, asall sub-pixel values of the reference frame required to determine amotion vector with the desired sub-pixel resolution are calculatedbefore-hand in block 730 and stored in frame store 740. Thus, they aredirectly available for use in the block-matching process and can beretrieved as required from frame store 740 by motion estimation block760. However, even in the case where all sub-pixel values are availablefrom frame store 740, it is still more computationally efficient toperform block-matching as a two-step process, as fewer differencecalculations are required. It should be appreciated that while fullbefore-hand sub-pixel value interpolation reduces computationalcomplexity in the encoder, it is not the most efficient approach interms of memory consumption.

In embodiments of the encoder in which both before-hand and on-demandsub-pixel value interpolation are used, motion estimation block 760 isimplemented in such a way that it can retrieve sub-pixel valuespreviously calculated in before-hand sub-pixel value interpolation block730 and stored in frame store 740 and further control on-demandsub-pixel value interpolation block 750 to calculate any additionalsub-pixel values that may be required. The block-matching process may beperformed as a one-step or a two-step process. If a two-stepimplementation is used, before-hand calculated sub-pixel valuesretrieved from frame store 740 may be used in the first step of theprocess and the second step may be implemented so as to use sub-pixelvalues calculated by on-demand sub-pixel value interpolation block 750.In this case, certain sub-pixel values used in the second step of theblock matching process may need to be calculated multiple times assuccessive comparisons are made, but the number of such duplicatecalculations is significantly less than if before-hand sub-pixel valuecalculation is not used. Furthermore, memory consumption is reduced withrespect to embodiments in which only before-hand sub-pixel valueinterpolation is used.

Once the motion estimation block 760 has produced a motion vector forthe macroblock of the current frame under examination, it outputs themotion vector to the motion field coding block 770. Motion field codingblock 770 then approximates the motion vector received from motionestimation block 760 using a motion model. The motion model generallycomprises a set of basis functions. More specifically, the motion fieldcoding block 770 represents the motion vector as a set of coefficientvalues (known as motion coefficients) which, when multiplied by thebasis functions, form an approximation of the motion vector. The motioncoefficients 724 are passed from motion field coding block 770 to motioncompensated prediction block 780. Motion compensated prediction block780 also receives the pixel/sub-pixel values of the best-matchingcandidate test region of the reference frame identified by motionestimation block 760. In FIG. 7, these values are shown to be passed vialine 729 from on-demand sub-pixel interpolation block 750. Inalternative embodiments of the invention, the pixel values in questionare provided from the motion estimation block 760 itself.

Using the approximate representation of the motion vector generated bymotion field coding block 770 and the pixel/sub-pixel values of thebest-matching candidate test region, motion compensated prediction block780 produces a macroblock of predicted pixel values. The macroblock ofpredicted pixel values represents a prediction for the pixel values ofthe current macroblock generated from the interpolated reference frame.The macroblock of predicted pixel values is passed to the combiner 716where it is subtracted from the new current frame in order to produceprediction error information 723 for the macroblock, as described in theforegoing.

The motion coefficients 724 formed by motion field coding block are alsopassed to the MUX/DMUX unit 790, where they are combined with predictionerror information 723 for the macroblock in question and possiblecontrol information from control manager 720 to form an encoded videostream 725 for transmission to a receiving terminal.

Operation of a video decoder 800 according to the invention will now bedescribed. Referring to FIG. 8, the decoder 800 comprises ademultiplexing unit (MUX/DMUX) 810, which receives the encoded videostream 725 from the encoder 700 and demultiplexes it, an inversequantiser 820, an inverse DCT transformer 830, a motion compensatedprediction block 840, a frame store 850, a combiner 860, a controlmanager 870, an output 880, before-hand sub-pixel value interpolationblock 845 and on-demand sub-pixel interpolation block 890 associatedwith the motion compensated prediction block 840. In practice thecontrol manager 870 of the decoder 800 and the control manager 720 ofthe encoder 700 may be the same processor. This may be the case if theencoder 700 and decoder 800 are part of the same video codec.

FIG. 8 shows an embodiment in which a combination of before-hand andon-demand sub-pixel value interpolation is used in the decoder. In otherembodiments, only before-hand sub-pixel value interpolation is used, inwhich case decoder 800 does not include on-demand sub-pixel valueinterpolation block 890. In a preferred embodiment of the invention, nobefore-hand sub-pixel value interpolation is used in the decoder andtherefore before-hand sub-pixel value interpolation block 845 is omittedfrom the decoder architecture. If both before-hand and on-demandsub-pixel value interpolation are performed, the decoder comprises bothblocks 845 and 890.

The control manager 870 controls the operation of the decoder 800 inresponse to whether an INTRA or an INTER frame is being decoded. AnINTRA/INTER trigger control signal, which causes the decoder to switchbetween decoding modes is derived, for example, from picture typeinformation provided in the header portion of each compressed videoframe received from the encoder. The INTRA/INTER trigger control signalis passed to control manager 870 via control line 815, together withother video codec control signals demultiplexed from the encoded videostream 725 by the MUX/DMUX unit 810.

When an INTRA frame is decoded, the encoded video stream 725 isdemultiplexed into INTRA coded macroblocks and control information. Nomotion vectors are included in the encoded video stream 725 for an INTRAcoded frame. The decoding process is performed macroblock-by-macroblock.When the encoded information 723 for a macroblock is extracted fromvideo stream 725 by MUX/DMUX unit 810, it is passed to inverse quantiser820. The control manager controls inverse quantiser 820 to apply asuitable level of inverse quantisation to the macroblock of encodedinformation, according to control information provided in video stream725. The inverse quantised macroblock is then inversely transformed inthe inverse DCT transformer 830 to form a decoded block of imageinformation. Control manager 870 controls combiner 860 to prevent anyreference information being used in the decoding of the INTRA codedmacroblock. The decoded block of image information is passed to thevideo output 880 of the decoder.

In embodiments of the decoder which employ before-hand sub-pixel valueinterpolation, the decoded block of image information (i.e. pixelvalues) produced as a result of the inverse quantisation and inversetransform operations performed in blocks 820 and 830 is passed tobefore-hand sub-pixel value interpolation block 845. Here, sub-pixelvalue interpolation is performed according to the method of theinvention, the degree of before-hand sub-pixel value interpolationapplied being determined by the details of the decoder implementation.In embodiments of the invention in which on-demand sub-pixel valueinterpolation is not performed, before-hand sub-pixel valueinterpolation block 845 interpolates all sub-pixelvalues. In embodimentsthat use a combination of before-hand and on-demand sub-pixel valueinterpolation, before-hand sub-pixel value interpolation block 845interpolates a certain sub-set of sub-pixel values. This may comprise,for example, all sub-pixels at half-pixel locations, or a combination ofsub-pixels at half-pixel and one quarter-pixel locations. In any case,after before-hand sub-pixel value interpolation, the interpolatedsub-pixel values are stored in frame store 850, together with theoriginal decoded pixel values. As subsequent macroblocks are decoded,before-hand interpolated and stored, a decoded frame, at least partiallyinterpolated to sub-pixel resolution is progressively assembled in theframe store 850 and becomes available for use as a reference frame formotion compensated prediction.

In embodiments of the decoder which do not employ before-hand sub-pixelvalue interpolation, the decoded block of image information (i.e. pixelvalues) produced as a result of the inverse quantisation and inversetransform operations performed on the macroblock in blocks 820 and 830is passed directly to frame store 850. As subsequent macroblocks aredecoded and stored, a decoded frame, having unit pixel resolution isprogressively assembled in the frame store 850 and becomes available foruse as a reference frame for motion compensated prediction.

When an INTER frame is decoded, the encoded video stream 725 isdemultiplexed into encoded prediction error information 723 for eachmacroblock of the frame, associated motion coefficients 724 and controlinformation. Again, the decoding process is performedmacroblock-by-macroblock. When the encoded prediction error information723 for a macroblock is extracted from the video stream 725 by MUX/DMUXunit 810, it is passed to inverse quantiser 820. Control manager 870controls inverse quantiser 820 to apply a suitable level of inversequantisation to the macroblock of encoded prediction error information,according to control information received in video stream 725. Theinverse quantised macroblock of prediction error information is theninversely transformed in the inverse DCT transformer 830 to yielddecoded prediction error information for the macroblock.

The motion coefficients 724 associated with the macroblock in questionare extracted from the video stream 725 by MUX/DMUX unit 810 and passedto motion compensated prediction block 840, which reconstructs a motionvector for the macroblock using the same motion model as that used toencode the INTER-coded macroblock in encoder 700. The reconstructedmotion vector approximates the motion vector originally determined bymotion estimation block 760 of the encoder. The motion compensatedprediction block 840 of the decoder uses the reconstructed motion vectorto identify the location of a block of pixel/sub-pixel values in aprediction reference frame stored in frame store 850. The referenceframe may be, for example, a previously decoded INTRA frame, or apreviously decoded INTER frame. In either case, the block ofpixel/sub-pixel values indicated by the reconstructed motion vector,represents the prediction of the macroblock in question.

The reconstructed motion vector may point to any pixel or sub-pixel. Ifthe motion vector indicates that the prediction for the currentmacroblock is formed from pixel values (i.e. the values of pixels atunit pixel locations), these can simply be retrieved from frame store850, as the values in question are obtained directly during the decodingof each frame. If the motion vector indicates that the prediction forthe current macroblock is formed from sub-pixel values, these musteither be retrieved from frame store 850, or calculated in on-demandsub-pixel interpolation block 890. Whether sub-pixel values must becalculated, or can simply be retrieved from the frame store, depends onthe degree of before-hand sub-pixel value interpolation used in thedecoder.

In embodiments of the decoder that do not employ before-hand sub-pixelvalue interpolation, the required sub-pixel values are all calculated inon-demand sub-pixel value interpolation block 890. On the other hand, inembodiments in which all sub-pixel values are interpolated before-hand,motion compensated prediction block 840 can retrieve the requiredsub-pixel values directly from the frame store 850. In embodiments thatuse a combination before-hand and on-demand sub-pixel valueinterpolation, the action required in order to obtain the requiredsub-pixel values depends on which sub-pixel values are interpolatedbefore-hand. Taking as an example an embodiment in which all sub-pixelvalues at half-pixel locations are calculated before-hand, it is evidentthat if a reconstructed motion vector for a macroblock points to a pixelat unit location or a sub-pixel at half-pixel location, all the pixel orsub-pixel values required to form the prediction for the macroblock arepresent in the frame store 850 and can be retrieved from there by motioncompensated prediction block 840. If, however, the motion vectorindicates a sub-pixel at a quarter-pixel location, the sub-pixelsrequired to form the prediction for the macroblock are not present inframe store 850 and are therefore calculated in on-demand sub-pixelvalue interpolation block 890. In this case, on-demand sub-pixel valueinterpolation block 890 retrieves any pixel or sub-pixel required toperform the interpolation from frame store 850 and applies theinterpolation method described below. Sub-pixel values calculated inon-demand sub-pixel value interpolation block 890 are passed to motioncompensated prediction block 840.

Once a prediction for a macroblock has been obtained, the prediction(that is, a macroblock of predicted pixel values) is passed from motioncompensated prediction block 840 to combiner 860 where it combined withthe decoded prediction error information for the macroblock to form areconstructed image block which, in turn, is passed to the video output880 of the decoder.

It should be appreciated that in practical implementations of encoder700 and decoder 800, the extent to which frames are before-handsub-pixel interpolated, and thus the amount of on-demand sub-pixel valueinterpolation that is performed, can be chosen according to, or dictatedby, the hardware implementation of the video encoder 700, or theenvironment in which it is intended to be used. For example, if thememory available to the video encoder is limited, or memory must bereserved for other functions, it is appropriate to limit the amount ofbefore-hand sub-pixel value interpolation that is performed. In othercases, where the microprocessor performing the video encoding operationhas limited processing capacity, e.g. the number of operations persecond that can be executed is comparatively low, it is more appropriateto restrict the amount of on-demand sub-pixel value interpolation thatis performed. In a mobile communications environment, for example, whenvideo encoding and decoding functionality is incorporated in a mobiletelephone or similar wireless terminal for communication with a mobiletelephone network, both memory and processing power may be limited. Inthis case a combination of before-hand and on-demand sub-pixel valueinterpolation may be the best choice to obtain an efficientimplementation in the video encoder. In video decoder 800, use ofbefore-hand sub-pixel value is generally not preferred, as it typicallyresults in the calculation of many sub-pixel values that are notactually used in the decoding process. However, it should be appreciatedthat although different amounts of before-hand and on-demandinterpolation can be used in the encoder and decoder in order tooptimise the operation of each, both encoder and decoder can beimplemented so as to use the same division between before-hand andon-demand sub-pixel value interpolation.

Although the foregoing description does not describe the construction ofbi-directionally predicted frames (B-frames) in the encoder 700 and thedecoder 800, it should be understood that in embodiments of theinvention, such a capability may be provided. Provision of suchcapability is considered within the ability of one skilled in the art.

An encoder 700 or a decoder 800 according to the invention can berealised using hardware or software, or using a suitable combination ofboth. An encoder or decoder implemented in software may be, for example,a separate program or a software building block that can be used byvarious programs. In the above description and in the drawings, thefunctional blocks are represented as separate units, but thefunctionality of these blocks can be implemented, for example, in onesoftware program unit.

The encoder 700 and decoder 800 can further be combined in order to forma video codec having both encoding and decoding functionality. Inaddition to being implemented in a multimedia terminal, such a codec mayalso be implemented in a network. A codec according to the invention maybe a computer program or a computer program element, or it may beimplemented at least partly using hardware.

The sub-pixel interpolation method used in the encoder 700 and decoder800 according to the invention now be described in detail. The methodwill first be introduced at a general conceptual level and then twopreferred embodiments will be described. In the first preferredembodiment, sub-pixel value interpolation is performed to ¼ pixelresolution and in the second the method is extended to/pixel resolution.

It should be noted that interpolation must produce identical values inthe encoder and the decoder, but its implementation should be optimizedfor both entities separately. For example, in an encoder according tothe first embodiment of the invention in which sub-pixel valueinterpolation is preformed to ¼ pixel resolution, it is most efficientto calculate ½ resolution pixels before-hand and to calculate values for¼ resolution sub-pixels in an on-demand fashion, only when they areneeded during motion estimation. This has the effect of limiting memoryusage while keeping the computational complexity/burden at an acceptablelevel. In the decoder, on the other hand, it is advantageous not topre-calculate any of the sub-pixels. Therefore, it should be appreciatedthat a preferred embodiment of the decoder does not include before-handsub-pixel value interpolation block 845 and all sub-pixel valueinterpolation is performed in on-demand sub-pixel value interpolationblock 890.

In the description of the interpolation method provided below,references are made to the pixel positions depicted in FIG. 14 a. Inthis figure pixels labelled A represent original pixels (that is, pixelsresiding at unit horizontal and vertical locations). Pixels labelledwith other letters represent sub-pixels that are to be interpolated. Thedescription that follows will adhere to the previously introducedconventions regarding the description of pixel and sub-pixel locations.

Next, the steps required to interpolate all sub-pixel positions aredescribed:

Values for the ½ resolution sub-pixels labelled b are obtained by firstcalculating an intermediate value b using a Kth order filter, accordingto: $\begin{matrix}{b = {\sum\limits_{i = 1}^{K}{x_{i}A_{i}}}} & (9)\end{matrix}$where x_(i) is a vector of filter coefficients, A_(i) is a correspondingvector of original pixel values A situated at unit horizontal and unitvertical locations, and K is an integer which defines the order of thefilter. Thus, equation 9 can be re-expressed as:b=x ₁ A ₁ +x ₂ A ₂ +x ₃ A ₃ + . . . +x _(K−1) A _(K−1) +x _(K) A_(K)  (10)

The values of the filter coefficients x_(i) and the order of the filterK may vary from embodiment to embodiment. Equally, different coefficientvalues may be used in the calculation of different sub-pixels within anembodiment. In other embodiments, the values of filter coefficientsx_(i) and the order of the filter may depend on which of the ½resolution b sub-pixels is being interpolated. Pixels A_(i) are disposedsymmetrically with respect to the ½ resolution sub-pixel b beinginterpolated and are the closest neighbours of that sub-pixel. In thecase of the ½ resolution sub-pixel b situated at half horizontal andunit vertical location, pixels A_(i) are disposed horizontally withrespect to b (as shown in FIG. 14 b). If the ½ resolution sub-pixel bsituated at unit horizontal and half vertical location is beinginterpolated, pixels A_(i) are disposed vertically with respect to b (asshown in FIG. 14 c).

A final value for ½ resolution sub-pixel b is calculated by dividingintermediate value b by a constant scale₁, truncating it to obtain aninteger number and clipping the result to lie in the range [0, 2^(n)−1].In alternative embodiments of the invention rounding may be performedinstead of truncation. Preferably, constant scale₁ is chosen to be equalto the sum of filter coefficients x_(i).

A value for the ½ resolution sub-pixel labelled c is also obtained byfirst calculating an intermediate value c using an Mth order filter,according to: $\begin{matrix}{c = {\sum\limits_{i = 1}^{M}{y_{i}b_{i}}}} & (11)\end{matrix}$where y_(i) is a vector of filter coefficients, b_(i) is a correspondingvector of intermediate values b_(i) in the horizontal or verticaldirection. i.e.:c=y ₁ b ₁ +y ₂ b ₂ +y ₃ b ₃ + . . . +y _(M−1) b _(M−1) +y _(M) b_(M)  (12)

The values of the filter coefficients y_(i) and the order of the filterM may vary from embodiment to embodiment. Equally, different coefficientvalues may be used in the calculation of different sub-pixels within anembodiment. Preferably, the b values are intermediate values for ½resolution sub-pixels b which are disposed symmetrically with respect to½ resolution sub-pixel c and are the closest neighbours of sub-pixel c.In an embodiment of the invention, the ½ resolution sub-pixels b aredisposed horizontally with respect to sub-pixel c, in an alternativeembodiment they are disposed vertically with respect to sub-pixel c.

A final value of ½ resolution sub-pixel c is computed by dividingintermediate value c by a constant scale₂, truncating it to obtain aninteger number and clipping the result to lie in the range [0, 2^(n)−1].In alternative embodiments of the invention rounding may be performedinstead of truncation. Preferably, constant scale₂ is equal toscale₁*scale₁.

It should be noted that the use of intermediate values b in thehorizontal direction leads to the same result as using intermediatevalues b in the vertical direction.

There are two alternatives for interpolating values for the ¼ resolutionsub-pixels labelled h. Both involve linear interpolation along adiagonal line linking ½ resolution sub-pixels neighbouring the ¼resolution sub-pixel h being interpolated. In a first embodiment, avalue for sub-pixel h is calculated by averaging the values of the two ½resolution sub-pixels b closest to sub-pixel h. In a second embodiment,a value for sub-pixel h is calculated by averaging the values of theclosest pixel A and the closest ½ resolution sub-pixel c. It should beappreciated that this provides the possibility of using differentcombinations of diagonal interpolations to determine the values forsub-pixels h within the confines of different groups of 4 image pixelsA. However, it should also be realised that the same combination shouldbe used in both the encoder and the decoder in order to produceidentical interpolation results. FIG. 15 depicts 4 possible choices ofdiagonal interpolation for sub-pixels h in adjacent groups of 4 pixelswithin an image. Simulations in the TML environment have verified thatboth embodiments result in similar compression efficiency. The secondembodiment has higher complexity, since calculation of sub-pixel crequires calculation of several intermediate values. Therefore the firstembodiment is preferred.

Values for ¼ resolution sub-pixels labelled d and g are calculated fromthe values of their nearest horizontal neighbours using linearinterpolation. In other words, a value for ¼ resolution sub-pixel d isobtained by averaging values of its nearest horizontal neighbours,original image pixel A and ½ resolution sub-pixel b. Similarly, a valuefor ¼ resolution sub-pixel g is obtained by taking the average of itstwo nearest horizontal neighbours, ½ resolution sub-pixels b and c.

Values for ¼ resolution sub-pixels labelled e, f and i are calculatedfrom the values of their nearest neighbours in the vertical directionusing linear interpolation. More specifically, a value for ¼ resolutionsub-pixel e is obtained by averaging the values of its two nearestvertical neighbours, original image pixel A and ½ resolution sub-pixelb. Similarly, a value for ¼ resolution sub-pixel f is obtained by takingthe average of its two nearest vertical neighbours, ½ resolutionsub-pixels b and c. In an embodiment of the invention, a value for ¼resolution sub-pixel i is obtained in manner identical to that justdescribed in connection with ¼ resolution sub-pixel f. However, in analternative embodiment of the invention, and in common with the H.26test models TML5 and TML6 previously described, ¼ resolution sub-pixel iis determined using the values of the four closest original imagepixels, according to (A₁+A₂+A₃+A₄+2)/4.

It should also be noted that in all cases where an average involvingpixel and/or sub-pixel values is determined, the average may be formedin any appropriate manner. For example, the value for ¼ resolutionsub-pixel d can be defined as d=(A+b)/2 or as d=(A+b+1)/2. The additionof 1 to the sum of values for pixel A and ½ resolution sub-pixel b hasthe effect of causing any rounding or truncation operation subsequentlyapplied to round or truncate the value for d to the next highest integervalue. This is true for any sum of integer values and may be applied toany of the averaging operations performed according to the method of theinvention in order to control rounding or truncation effects.

It should be noted that the sub-pixel value interpolation methodaccording to the invention provides advantages over each of TML5 andTML6.

In contrast to TML5, in which the values of some of the ¼ resolutionsub-pixels depend on previously interpolated values obtained for other ¼resolution sub-pixels, in the method according to the invention, all ¼resolution sub-pixels are calculated from original image pixels or ½resolution sub-pixel positions using linear interpolation. Thus, thereduction in precision of those ¼ resolution sub-pixel values whichoccurs in TML5 due to the intermediate truncation and clipping of theother ¼ resolution sub-pixels from which they are calculated, does nottake place in the method according to the invention. In particular,referring to FIG. 14 a, ¼ resolution sub-pixels h (and sub-pixel i inone embodiment of the invention) are interpolated diagonally in order toreduce dependency on other ¼-pixels. Furthermore, in the methodaccording to the invention, the number of calculations (and thereforethe number of processor cycles) required to obtain a value for those ¼resolution sub-pixels in the decoder is reduced compared with TML5.Additionally, the calculation of any ¼ resolution sub-pixel valuerequires a number of calculations which is substantially similar to thenumber of calculations required to determine any other ¼ resolutionsub-pixel value. More specifically, in a situation where the required ½resolution sub-pixel values are already available, e.g. they have beencalculated before-hand, the number of calculations required tointerpolate a ¼ resolution sub-pixel value from the pre-calculated ½resolution sub-pixel values is the same as the number of calculationsrequired to calculate any other ¼ resolution sub-pixel value from theavailable ½ resolution sub-pixel values.

In comparison with TML6, the method according to the invention does notrequire high precision arithmetic to be used in the calculation of allsub-pixels. Specifically, as all of the ¼ resolution sub-pixel valuesare calculated from original image pixels or ½ resolution sub-pixelvalues using linear interpolation, lower precision arithmetic can beused in their interpolation. Consequently, in hardware implementationsof the inventive method, for example in an ASIC (Application SpecificIntegrated Circuit), the use of lower precision arithmetic reduces thenumber of components (e.g. gates) that must be devoted to thecalculation of ¼ resolution sub-pixel values. This, in turn, reduces theoverall area of silicon that must be dedicated to the interpolationfunction. As the majority of sub-pixels are, in fact, 1/4 resolutionsub-pixels (12 out of the 15 sub-pixels illustrated in FIG. 14 a), theadvantage provided by the invention in this respect is particularlysignificant. In software implementations, where sub-pixel interpolationis performed using the standard instruction set of a general purpose CPU(Central Processor Unit) or using a DSP (Digital Signal Processor), areduction in the precision of the arithmetic required generally leads toan increase in the speed at which the calculations can be performed.This is particularly advantageous in ‘low cost’ implementations, inwhich it is desirable to use a general purpose CPU rather than any formof ASIC.

The method according to the invention provides still further advantagescompared with TML5. As mentioned previously, in the decoder only 1 outof the 15 sub-pixel positions is required at any given time, namely thatwhich is indicated by received motion vector information. Therefore, itis advantageous if the value of a sub-pixel in any sub-pixel locationcan be calculated with the minimum number of steps that result in acorrectly interpolated value. The method according to the inventionprovides this capability. As mentioned in the detailed descriptionprovided above, ½ resolution sub-pixel c can be interpolated byfiltering in either the vertical or the horizontal direction, the samevalue being obtained for c regardless of whether horizontal or verticalfiltering is used. The decoder can therefore take advantage of thisproperty when calculating values for ¼ resolution sub-pixels f and g insuch a way as to minimise the number of operations required in order toobtain the required values. For example, if the decoder requires a valuefor ¼ resolution sub-pixel f, ½ resolution sub-pixel c shouldinterpolated in the vertical direction. If a value is required for ¼resolution sub-pixel g, it is advantageous to interpolate a value for cin the horizontal direction. Thus, in general, it can be said that themethod according to the invention provides flexibility in the way inwhich values are derived for certain ¼ resolution sub-pixels. No suchflexibility is provided in TML5.

Two specific embodiments will now be described in detail. The firstrepresents a preferred embodiment for calculating sub-pixels with up to¼ pixel resolution, while in the second, the method according to theinvention is extended to the calculation of values for sub-pixels havingup to ⅛ pixel resolution. For both embodiments a comparison is providedbetween the computational complexity/burden resulting from use of themethod according to the invention and that which would result from useof the interpolation methods according to TML5 and TML6 in equivalentcircumstances.

The preferred embodiment for interpolating sub-pixels at ¼ pixelresolution will be described with reference to FIGS. 14 a, 14 b and 14c. In the following, it will be assumed that all image pixels and finalinterpolated values for sub-pixels are represented with 8-bits.

Calculation of ½ resolution sub-pixels at i) half unit horizontal andunit vertical locations and ii) unit horizontal and half unit verticallocations.

-   1. A value for the sub-pixel at half unit horizontal and unit    vertical location, that is ½ resolution sub-pixel b in FIG. 14 a, is    obtained by first calculating intermediate value    b=(A₁−5A₂+20A₃+20A₄−5A₅+A₆) using the values of the six pixels (A₁    to A₆) which are situated at unit horizontal and unit vertical    locations in either the row or the column of pixels containing b and    which are disposed symmetrically about b, as shown in FIGS. 14 b and    14 c. A final value for ½ resolution sub-pixel b is calculated as    (b+16)/32 where the operator / denotes division with truncation. The    result is clipped to lie in the range [0, 255].

Calculation of ½ resolution sub-pixels at half unit horizontal and halfunit vertical locations.

-   2. A value for the sub-pixel at half unit horizontal and half unit    vertical location, that is ½ resolution sub-pixel c in FIG. 14 a, is    calculated as c=(b₁−5b₂+20b₃+20b₄−5b₅+b₆+512)/1024 using the    intermediate values b for the six closest ½ resolution sub-pixels    which are situated in either the row or the column of sub-pixels    containing c and which are disposed symmetrically about c. Again,    operator / denotes division with truncation and the result is    clipped to lie in the range [0, 255]. As previously explained, using    intermediate values b for ½ resolution sub-pixels b in the    horizontal direction leads to the same result as using intermediate    values b for ½ resolution sub-pixels b in the vertical direction.    Thus, in an encoder according to the invention, the direction for    interpolating ½ resolution sub-pixels b can be chosen according to a    preferred mode of implementation. In a decoder according to the    invention, the direction for interpolating sub-pixels b is chosen    according to which, if any, ¼ resolution sub-pixels will be    interpolated using the result obtained for ½ resolution sub-pixel c.

Calculation of ¼ resolution sub-pixels at i) quarter unit horizontal andunit vertical locations; ii) quarter unit horizontal and half unitvertical locations; iii) unit horizontal and quarter unit verticallocations; and iv) half unit horizontal and quarter unit verticallocations.

-   3. Values for ¼ resolution sub-pixels d, situated at quarter unit    horizontal and unit vertical locations are calculated according to    d=(A+b)/2 using the nearest original image pixel A and the closest ½    resolution sub-pixel b in the horizontal direction. Similarly,    values for ½ resolution sub-pixels g, situated at quarter unit    horizontal and half unit vertical locations are calculated according    to g=(b+c)/2 using the two nearest ½ resolution sub-pixels in the    horizontal direction. In a similar manner, values for ¼ resolution    sub-pixels e, situated at unit horizontal and quarter unit vertical    locations, are calculated according e=(A+b)/2 using the nearest    original image pixel A and the closest ½ resolution sub-pixel b in    the vertical direction. Values for ¼ resolution sub-pixels f,    situated at half unit horizontal and quarter unit vertical    locations, are determined from f=(b+c)/2 using the two nearest ½    resolution sub-pixel in the vertical direction. In all cases,    operator / denotes division with truncation.

Calculation of ¼ resolution sub-pixels at quarter unit horizontal andquarter unit vertical locations.

-   4. Values for ¼ resolution sub-pixels h, situated at quarter unit    horizontal and quarter unit vertical locations are calculated    according to h=(b₁+b₂)/2, using the two nearest ½ resolution    sub-pixels b in the diagonal direction. Again, operator / denotes    division with truncation.-   5. A value for the ¼ resolution sub-pixel labeled i is computed from    i=(A₁+A₂+A₃+A₄+2)/4 using the four nearest original pixels A. Once    more, operator / denotes division with truncation.

An analysis of the computational complexity of the first preferredembodiment of the invention will now be presented.

In the encoder, it is likely that the same sub-pixel values will becalculated multiple times. Therefore, and as previously explained, thecomplexity of the encoder can be reduced by pre-calculating allsub-pixel values and storing them in memory. However, this solutionincreases memory usage by a large margin. In the preferred embodiment ofthe invention, in which motion vector accuracy is ¼ pixel resolution inboth the horizontal and vertical dimensions, storing pre-calculatedsub-pixel values for the whole image requires 16 times the memoryrequired to store the original, non-interpolated image. To reduce memoryusage, all ½ resolution sub-pixels can be interpolated before-hand and ¼resolution sub-pixels can be calculated on-demand, that is, only whenthey are needed. According to the method of the invention, on-demandinterpolation of values for ¼ resolution sub-pixels only requires linearinterpolation from ½ resolution sub-pixels. Four times the originalpicture memory is required to store the pre-calculated ½ resolutionsub-pixels since only 8 bits are necessary to represent them.

However, if the same strategy of pre-calculating all ½ resolutionsub-pixels using before-hand interpolation is used in conjunction withthe direct interpolation scheme of TML6, the memory requirementsincrease to 9 times the memory required to store the originalnon-interpolated image. This results from the fact that a larger numberof bits is required to store the high precision intermediate valuesassociated with each ½ resolution sub-pixel in TML6. In addition, thecomplexity of sub-pixel interpolation during motion estimation is higherin TML6, since scaling and clipping has to be performed for every ½ and¼ sub-pixel location.

In the following, the complexity of the sub-pixel value interpolationmethod according to the invention, when applied in a video decoder, iscompared with that of the interpolation schemes used in TML5 and TML6.Throughout the analysis which follows, it is assumed that in each methodthe interpolation of any sub-pixel value is performed using only theminimum number of steps required to obtain a correctly interpolatedvalue. It is further assumed that each method is implemented in a blockbased manner, that is, intermediate values common for all the sub-pixelsto be interpolated in a particular N×M block are calculated only once.An illustrative example is provided in FIG. 16. Referring to FIG. 16, itcan be seen that in order to calculate a 4×4 block of ½ resolutionsub-pixels c, a 9×4 block of ½ resolution sub-pixels b is firstcalculated.

Compared with the sub-pixel value interpolation method used in TML5, themethod according to the invention has a lower computational complexityfor the following reasons:

-   1. Unlike the sub-pixel value interpolation scheme used in TML5,    according to the method of the invention, a value for ½ resolution    sub-pixel c can be obtained by filtering in either the vertical or    the horizontal direction. Thus, in order to reduce the number of    operations, ½ resolution sub-pixel c can be interpolated in the    vertical direction if a value for ¼ resolution sub-pixel f is    required and in the horizontal direction if a value for ¼ resolution    sub-pixel pixel g is required. As an example, FIG. 17 shows all the    ½ resolution sub-pixel values that must be calculated in order to    interpolate values for ¼ resolution sub-pixels g in an image block    defined by 4×4 original image pixels using the interpolation method    of TML5 (FIG. 17 a) and using the method according to the invention    (FIG. 17 b). In this example, the sub-pixel value interpolation    method according to TML5 requires a total of 88½ resolution    sub-pixels to be interpolated, while the method according to the    invention requires the calculation of 72½ resolution sub-pixels. As    can be seen from FIG. 17 b, according to the invention, ½ resolution    sub-pixels c are interpolated in the horizontal direction in order    to reduce the number of calculations required.-   2. According to the method of the invention, ¼ resolution sub-pixel    h is calculated by linear interpolation from its two closest    neighbouring ½ resolution sub-pixels in the diagonal direction. The    respective numbers of ½ resolution sub-pixels that must be    calculated in order to obtain values for ¼ resolution sub-pixels h    within a 4×4 block of original image pixels using the sub-pixel    value interpolation method according to TML5 and the method    according to the invention are shown in FIGS. 18(a) and 18(b),    respectively. Using the method according to TML5 it is necessary to    interpolate a total of 56½ resolution sub-pixels, while according to    the method of the invention it is necessary to interpolate 40½    resolution sub-pixels.

Table 1 summarizes the decoder complexities of the three sub-pixel valueinterpolation methods considered here, that according to TML5, thedirect interpolation method used in TML6 and the method according to theinvention. Complexity is measured in terms of the number of 6-tapfiltering and linear interpolation operations performed. It is assumedthat Interpolation of ¼ resolution sub-pixel i is calculated accordingto i=(A₁+A₂+A₃+A₄+2)/4 which is bilinear interpolation and effectivelycomprises two linear interpolation operations. The operations needed tointerpolate sub-pixel values with one 4×4 block of original image pixelsare listed for each of the 15 sub-pixel positions which, for convenienceof reference, are numbered according to the scheme shown in FIG. 19.Referring to FIG. 19, location 1 is the location of an original imagepixel A and locations 2 to 16 are sub-pixel locations. Location 16 isthe location of ¼ resolution sub-pixel i. In order to compute theaverage number of operations it has been assumed that the probability ofa motion vector pointing to each sub-pixel position is the same. Theaverage complexity is therefore the average of the 15 sums calculatedfor each sub-pixel location and the single full-pixel location. TABLE 1Complexity of ¼ resolution sub-pixel interpolation in TML5, TML6 and themethod according to the invention. Inventive TML5 TML6 Method Locationlinear. 6-tap linear. 6-tap linear. 6-tap 1 0 0 0 0 0 0 3, 9 0 16 0 16 016 2, 4, 5, 13 16 16 0 16 16 16 11 0 52 0 52 0 52 7, 15 16 52 0 52 16 5210, 12 16 68 0 52 16 52 6, 8, 14 48 68 0 52 16 32 16 32 0 32 0 32 0Average 19 37 2 32 13 28.25

It can be seen from Table 1 that the method according to the inventionrequires fewer 6-tap filter operations than the sub-pixel valueinterpolation method according to TML6 and only a few additional linearinterpolation operations. Since 6-tap filter operations are much morecomplex than linear interpolation operations the complexity of the twomethods is similar. The sub-pixel value interpolation method accordingto TML5 has a considerably higher complexity.

The preferred embodiment for interpolating sub-pixels with up to ⅛ pixelresolution will now be described with reference to FIGS. 20, 21 and 22.

FIG. 20 presents the nomenclature used to describe pixels, ½ resolutionsub-pixels, ¼ resolution sub-pixels and ⅛ resolution sub-pixels in thisextended application of the method according to the invention.

-   1. Values for the ½ resolution and ¼ resolution sub-pixels labelled    b¹, b² and b³ in FIG. 20 are obtained by first calculating    intermediate values b¹=(−3A₁+12A₂−37A₃+229A₄+71A₅−21A₆+6A₇−A₈);    b²=(−3A₁+12A₂−39A₃+158A₄+158A₅−39A₆+12A₇−3A₈); and    b³=(−A₁+6A₂−21A₃+71A₄+229A₅−37A₆+13A₇−3A₈) using the values of the    eight nearest image pixels (A₁ to A₈) situated at unit horizontal    and unit vertical locations in either the row or the column    containing b¹, b² and b³ and disposed symmetrically about ½    resolution sub-pixel b². The asymmetries in the filter coefficients    used to obtain intermediate values b¹ and b³ account for the fact    that pixels A₁ to A₈ are not symmetrically located with respect to ¼    resolution sub-pixels b¹ and b³. Final values for sub-pixels b^(i),    i=1, 2, 3 are calculated according to b^(i)=(b^(i)+128)/256 where    the operator / denotes division with truncation. The result is    clipped to lie in the range [0, 255].-   2. Values for the ½ resolution and ¼ resolution sub-pixels labelled    c^(ij), i, j=1, 2, 3, are calculated according to c^(1j)=(−3 b₁    ^(j)+12b₂ ^(j)−37b₃ ^(j)+229b₄ ^(j)+71b₅ ^(j)−21b₆ ^(j)+6b₇ ^(j)−b₈    ^(j)+32768)/65536, c^(2j)=(−3b₁ ^(j)+12b₂ ^(j)−39b₃ ^(j)+158b₄    ^(j)+158b₅ ^(j)−39b₆ ^(j)+12b₇ ^(j)−3b₈ ^(j)+32768)/65536 and    c^(3j)=(−b₁ ^(j)+6b₂ ^(j)−21b₃ ^(j)+71b₄ ^(j)+229b₅ ^(j)−37b₆    ^(j)+13b₇ ^(j)−3b₈ ^(j)+32768)/65536 using the intermediate values    b¹, b² and b³ calculated for the eight closest sub-pixels (b₁ ^(j)    to b₈ ^(j)) in the vertical direction, sub-pixels b^(j) being    situated in the column comprising the ½ resolution and ¼ resolution    sub-pixels c^(ij) being interpolated and disposed symmetrically    about the ½ resolution sub-pixel c^(2j). The asymmetries in the    filter coefficients used to obtain values for sub-pixels c^(1j) and    c^(3j) account for the fact that sub-pixels b₁ ^(j) to b₈ ^(j) are    not symmetrically located with respect to ¼ resolution sub-pixels    c^(1j) and c^(3j). Once more, operator / denotes division with    truncation. Before the interpolated values for sub-pixels c^(ij) are    stored in the frame memory they are clipped to lie in the range [0,    255]. In an alternative embodiment of the invention, ½ resolution    and ¼ resolution sub-pixels c^(ij) are calculated using in an    analogous manner using intermediate values b¹, b² and b³ in the    horizontal direction.-   3. Values for ⅛ resolution sub-pixels labelled d are calculated    using linear interpolation from the values of their closest    neighbouring image pixel, ½ resolution or ¼ resolution sub-pixels in    the horizontal or vertical direction. For example, upper leftmost ⅛    resolution sub-pixel d is calculated according to d=(A+b¹+1)/2. As    before, operator / indicates division with truncation.-   4. Values for ⅛ resolution sub-pixels labelled e and f are    calculated using linear interpolation from the values of image    pixels, ½ resolution or ¼ resolution sub-pixels in the diagonal    direction. For example, referring to FIG. 20, upper leftmost pixel ⅛    resolution sub-pixel e is calculated according to e=(b¹+b¹+1)/2. The    diagonal direction to be used in the interpolation of each ⅛    resolution sub-pixel in a first preferred embodiment of the    invention, hereinafter referred to as ‘preferred method 1’, is    indicated in FIG. 21(a). Values for ⅛ resolution sub-pixels labelled    g are calculated according to g=(A+3c²²+3)/4. As always, operator /    denotes division with truncation. In an alternative embodiment of    the invention, hereinafter referred to as ‘preferred method 2’,    computational complexity is further reduced by interpolating ⅛    resolution sub-pixels f using linear interpolation from ½ resolution    sub-pixels b², that is, according to the relationship    f=(3b²+b²+2)/4. The b² sub-pixel which is closer to f is multiplied    by 3. The diagonal interpolation scheme used in this alternative    embodiment of the invention is depicted in FIG. 21(b). In further    alternative embodiments, different diagonal interpolation schemes    can be envisaged.

It should be noted that in all cases where an average involving pixeland/or sub-pixel values is used in the determination of ⅛ resolutionsub-pixels, the average may be formed in any appropriate manner. Theaddition of 1 to the sum of values used in calculating such an averagehas the effect of causing any rounding or truncation operationsubsequently applied to round or truncate the average in question to thenext highest integer value. In alternative embodiments of the invention,the addition of 1 is not used.

As in the case of sub-pixel value interpolation to ¼ pixel resolutionpreviously described, memory requirements in the encoder can be reducedby pre-calculating only a part of the sub-pixel values to beinterpolated. In the case of sub-pixel value interpolation to ⅛ pixelresolution, it is advantageous to calculate all ½ resolution and ¼resolution sub-pixels before-hand and to compute values for ⅛ resolutionsub-pixels in an on-demand fashion, only when they are required. Whenthis approach is taken, both the interpolation method according to TML5and that according to the invention require 16 times the originalpicture memory to store the ½ resolution and ¼ resolution sub-pixelvalues. However, if the direct interpolation method according to TML6 isused in the same way, intermediate values for the ½ resolution and ¼pixel resolution sub-pixels must be stored. These intermediate valuesare represented with 32-bit precision and this results in a memoryrequirement 64 times that for the original, non-interpolated image.

In the following, the complexity of the sub-pixel value interpolationmethod according to the invention, when applied in a video decoder tocalculate values for sub-pixels at up to ⅛ pixel resolution, is comparedwith that of the interpolation schemes used in TML5 and TML6. As in theequivalent analysis for ¼ pixel resolution sub-pixel value interpolationdescribed above, it is assumed that in each method the interpolation ofany sub-pixel value is performed using only the minimum number of stepsrequired to obtain a correctly interpolated value. It is also assumedthat each method is implemented in a block based manner, such thatintermediate values common for all the sub-pixels to be interpolated ina particular N×M block are calculated only once.

Table 2 summarizes complexities of the three interpolation methods.Complexity is measured in terms of the number of 8-tap filter and linearinterpolation operations performed in each method. The table presentsthe number of operations required to interpolate each of the 63⅛resolution sub-pixels within one 4×4 block of original image pixels,each sub-pixel location being identified with a corresponding number, asillustrated in FIG. 22. In FIG. 22, location 1 is the location of anoriginal image pixel and locations 2 to 64 are sub-pixel locations. Whencomputing the average number of operations, it has been assumed that theprobability of a motion vector pointing to each sub-pixel position isthe same. The average complexity is thus the average of the 63 sumscalculated for each sub-pixel location and the single full-pixellocation. TABLE 2 Complexity of ½ resolution sub-pixel interpolation inTML5, TML6 and the method according to the invention. (Results shownseparately for Preferred Method 1 and Preferred Method 2). PreferredPreferred TML5 TML6 Method 1 Method 2 Location linear. 8-tap linear.8-tap linera. 8-tap linear. 8-tap 1 0 0 0 0 0 0 0 0 3, 5, 7, 17, 33, 490 16 0 16 0 16 0 16 19, 21, 23, 35, 37, 39, 0 60 0 60 0 60 0 60 51, 53,55 2, 8, 9, 57 16 16 0 16 16 16 16 16 4, 6, 25, 41 16 32 0 16 16 32 1632 10, 16, 58, 64 32 76 0 60 16 32 16 32 11, 13, 15, 59, 61, 63 16 60 060 16 60 16 60 18, 24, 34, 40, 50, 56 16 76 0 60 16 60 16 60 12, 14, 60,62 32 120 0 60 16 32 16 32 26, 32, 42, 48 32 108 0 60 16 32 16 32 20,22, 36, 38, 52, 54 16 120 0 60 16 76 16 76 27, 29, 31, 43, 45, 47 16 760 60 16 76 16 76 28, 30, 44, 46 32 152 0 60 16 60 16 60 Average 64290.25 0 197.75 48 214.75 48 192.75

As can be seen from Table 2, the number of 8-tap filtering operationsperformed according to preferred methods 1 and 2 are, respectively, 26%and 34% lower than the number of 8-tap filtering operation performed inthe sub-pixel value interpolation method of TML5. The number of linearoperations is 25% lower, in both preferred method 1 and preferred method2, compared with TML5, but this improvement is of lesser importancecompared to the reduction in 8-tap filtering operations. It can furtherbe seen that the direct interpolation method used in TML6 has acomplexity comparable that of both preferred methods 1 and 2 when usedto interpolate values for ⅛ resolution sub-pixels.

In view of the foregoing description it will be evident to a personskilled in the art that various modifications may be made within thescope of the invention. While a number of preferred embodiments of theinvention have been described in detail, it should be apparent that manymodifications and variations thereto are possible, all of which fallwithin the true spirit and scope of the invention.

1. A method for sub-pixel value interpolation to determine values forsub-pixels situated within a rectangular bounded region defined by fourcorner pixels with no intermediate pixels between the corners, thepixels and sub-pixels being arranged in rows and columns, the pixel andsub-pixel locations being representable mathematically within therectangular bounded region using the co-ordinate notation K/2^(N),L/2^(N), K and L being positive integers having respective valuesbetween zero and 2^(N), N being a positive integer greater than one andrepresenting a particular degree of sub-pixel value interpolation, themethod comprising: interpolating a sub-pixel value for a sub-pixelhaving co-ordinates with odd values of both K and L, according to apredetermined choice of a weighted average of the value of anearest-neighbouring pixel and the value of the sub-pixel situated atco-ordinates ½, ½, and a weighted average of the values of a pair ofdiagonally-opposed sub-pixels having co-ordinates with even values ofboth K and L, including zero, situated within a quadrant of therectangular bounded region defined by corner pixels having co-ordinates½, ½ and the nearest neighbouring pixel; interpolating sub-pixel valuesfor sub-pixels having co-ordinates with K equal to an even value and Lequal zero and sub-pixels having co-ordinates with K equal to zero and Lequal to an even value, used in the interpolation of the sub-pixelshaving co-ordinates with odd values of both K and L, using weighted sumsof the values of pixels located in rows and columns respectively; andinterpolating sub-pixel values for sub-pixels having co-ordinates witheven values of both K and L, used in the interpolation of sub-pixelvalues for the sub-pixels having co-ordinates with odd values of both Kand L, using a predetermined choice of either a weighted sum of thevalues of sub-pixels having co-ordinates with K equal to an even valueand L equal to zero and the values of sub-pixels having correspondingco-ordinates in immediately adjacent rectangular bounded regions, or aweighted sum of the values of sub-pixels having co-ordinates with Kequal to zero and L equal to an even value and the values of sub-pixelshaving corresponding co-ordinates in immediately adjacent boundedrectangular regions.
 2. A method for quarter resolution sub-pixel valueinterpolation to determine values for sub-pixels situated within arectangular bounded region defined by four corner pixels with nointermediate pixels between the corners, the pixels and sub-pixels beingarranged in rows and columns, the pixel and sub-pixel locations beingrepresentable mathematically within the rectangular bounded region usingthe co-ordinate notation K/4, L/4, K and L being positive integershaving respective values between zero and 4, the method comprising:interpolating a sub-pixel value for a sub-pixel having co-ordinates withboth K and L equal to 1 or 3, according to a predetermined choice of aweighted average of the value of a nearest-neighbouring pixel and thevalue of the sub-pixel situated at co-ordinates 2/4, 2/4, and a weightedaverage of the values of a pair of diagonally-opposed sub-pixels havingco-ordinates with both K and L equal to zero, 2 or 4, situated within aquadrant of the rectangular bounded region defined by corner pixelshaving co-ordinates 2/4, 2/4 and the nearest neighbouring pixel;interpolating sub-pixel values for sub-pixels having co-ordinates with Kequal to 2 and L equal zero and sub-pixels having co-ordinates with Kequal to zero and L equal to 2, used in the interpolation of thesub-pixels having co-ordinates with both K and L equal to 1 or 3, usingweighted sums of the values of pixels located in rows and columnsrespectively; and interpolating a sub-pixel value for the sub-pixelhaving co-ordinates with both K and L equal to 2, used in theinterpolation of sub-pixel values for the sub-pixels having co-ordinateswith both K and L equal to 1 or 3, using a predetermined choice ofeither a weighted sum of the values of the sub-pixel having co-ordinateswith K equal to 2 and L equal to zero and the values of sub-pixelshaving corresponding co-ordinates in immediately adjacent rectangularbounded regions, or a weighted sum of the value of the sub-pixel havingco-ordinates with K equal to zero and L equal to 2 and the values ofsub-pixels having corresponding co-ordinates in immediately adjacentrectangular bounded regions.
 3. A method for eighth resolution sub-pixelvalue interpolation to determine values for sub-pixels situated within arectangular bounded region defined by four corner pixels with nointermediate pixels between the corners, the pixels and sub-pixels beingarranged in rows and columns, the pixel and sub-pixel locations beingrepresentable mathematically with the rectangular bounded region usingthe co-ordinate notation K/8, L/8, K and L being positive integershaving respective values between zero and 8, the method comprising:interpolating a sub-pixel value for a sub-pixel having co-ordinates withboth K and L equal to 1, 3, 5 or 7, according to a predetermined choiceof a weighted average of the value of a nearest-neighbouring pixel andthe value of the sub-pixel situated at co-ordinates 4/8, 4/8, and aweighted average of the values of a pair of diagonally-opposedsub-pixels having co-ordinates with both K and L equal to zero, 2, 4, 6or 8, situated within a quadrant of the rectangular bounded regiondefined by corner pixels having co-ordinates 4/8, 4/8 and the nearestneighbouring pixel; interpolating sub-pixel values for sub-pixels havingco-ordinates with K equal to 2, 4 or 6 and L equal zero and sub-pixelshaving co-ordinates with K equal to zero and L equal to 2, 4 or 6, usedin the interpolation of the sub-pixels having co-ordinates with both Kand L equal to 1, 3, 5 or 7, using weighted sums of the values of pixelslocated in rows and columns respectively; and interpolating sub-pixelvalues for sub-pixels having co-ordinates with both K and L equal to 2,4 or 6, used in the interpolation of sub-pixel values for the sub-pixelshaving co-ordinates with both K and L equal to 1, 3, 5 or 7, using apredetermined choice of either a weighted sum of the values ofsub-pixels having co-ordinates with K equal to 2, 4 or 6 and L equal tozero and the values of sub-pixels having corresponding co-ordinates inimmediately adjacent rectangular bounded regions or a weighted sum ofthe values of sub-pixels having co-ordinates with K equal to zero and Lequal to 2, 4 or 6 and the values of sub-pixels having correspondingco-ordinates in immediately adjacent rectangular bounded regions.